Set up TriBITS testing using GitHub Actions

[bartlettroscoe]

CC: @keita, @fnrizzi, @marcinwrobel1986, @MikolajZuzek

With a lot of work on TriBITS going on and coming up (i.e. #63, #299, etc.) we need solid testing for TriBITS. I have solid testing of TriBITS on local machines set up using the checkin-test.py script (see checkin-test-crf450.sh and those it calls) but we really need to have this for PR testing as well. In the past, I did that with Travis CI. But for reasons that I can't go into here, Travis CI is not a viable option going forward.

The obvious choice for this would seem to be to use GitHub Actions (GHA) for this. But I don't know anything about GHA so this will be a learning curve.

For dependencies I need:

• CMake 3.17
• C++, C, and Fortran compilers
• Python 2.7

Other dependencies that would be nice to have:

• CMake current release (current CMake 3.19 but soon to be 3.20)
• Python 3.x
• Ninja 1.10+

And this would post to CDash at:

• https://testing.sandia.gov/cdash/viewSubProjects.php?project=TriBITS

At a minimum, there needs to be a build with:

• CMake 3.17 + Python 2.7 (or 3.x)

Other builds that would be nice to have include:

• CMake current + Python 3.x (or 2.7 if base build is using 3.x)
• CMake 3.17 + Ninja 1.10
• CMake 3.17 + Ninja 1.10 (no Fortran)

Tasks:

• Scope out new GitHub Actions driver (see PRs CI: draft action yaml #366 and TriBITS CI testing with GitHub Actions (#363) #389)
• Create new branch '363-github-actions-tests' based off '377-provide-feedback-on-TriBITS-package-dependencies-data-structures-CI', rebase, clean up commits and create new PR .
• Add install of docutils to github actions job and enable build_docs test.
• Create file tribits_cmake-3.21.0_makefiles_python-3.8_g++-9_tweaks.cmake disables for cmake-3.21.0 GHA build to disable tests that fail with CMake 3.20.
• Add new triggers workflow_dispatch and schedule.
• Determine what branch the GitHub Action jobs was invoked on and send jobs to CDash to different CDash groups based on the event trigger: PRs and manual triggers to 'Experimental', pushes to 'master' to 'Continuous', cron jobs to 'Nightly' (see GITHUB_EVENT_NAME).
• Set up TriBITS CDash site to send out emails if there are any failures reported to 'Continuous' or 'Nightly' dashboard'.
• Merge new PR branch.
• Create a side branch '363-github-actions-tests-show-ctest-3.21-defects' and create a PR that demonstrates the failures with CMake 3.21 that don't occur with CMake 3.17, post SNL Kitware issue ... See PR DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (#363) #394 and SNL Kitware #184.
• Do a push to 'master' and verify that the builds are fired off and submit to the 'Continuous' CDash group.
• Verify that the workflow_dispatch trigger works and submits to the Experimental CDash group.
• Verify that the schedule trigger works and submits to the 'Nightly' CDash group.
• Add pull request number to the build names and send results to CDash "Pull Request" Group for PR builds.
• Print ninja version PR Print the ninja path and version (#363) #397
• Fix problem with pr-null_ in build name of push testing to master ... PR Deal with pr null in not preprending build name (#363) #398
• Update TriBITS GitHub wiki from Travis CI to GitHub Actions setup ... see here and here.

[bartlettroscoe]

As part of the IDEAS-ECP project, I have been looking into GHA so it seems like a good time to try setting up TriBITS testing (and posting to CDash) using GHA.

Some good starting references I was pointed to were:

• https://itnext.io/getting-started-with-github-actions-fe94167dbc6d
• https://gist.github.com/br3ndonland/f9c753eb27381f97336aa21b8d932be6

Those link to other pages with lots more information.

Some info on doing C++ testing with CMake can be found at:

• https://cristianadam.eu/20191222/using-github-actions-with-c-plus-plus-and-cmake/

That shows how to test with CMake + Ninja and does so on Linux and Windows! This will take some time to adapt, but it should do it hopefully.

[bartlettroscoe]

@keita, @fnrizzi, @marcinwrobel1986, @MikolajZuzek,

Any of you guys have GitHub Actions experience that you could use to help set up automated testing on GitHub for TriBITS to help drive the TriBITS refactoring work?

[fnrizzi]

@bartlettroscoe
Hi Ross! I have some, but not extensive (most of my GA experience relates to C++ and Python testing for this project https://github.com/Pressio).
I just connected with @marcinwrobel1986 @MikolajZuzek: they said they know GA a bit too. We also have another person in our group that knows it too. I would say we know enough that we can help out getting started.
Speaking of this, Keita mentioned that he would setup a meeting between all of us for mid April to talk about the trilinos work. What if we try to schedule that soon and use to talk about this too and how this fits the current work agreement we have?

[bartlettroscoe]

I have some, but not extensive (most of my GA experience relates to C++ and Python testing for this project https://github.com/Pressio).

@fnrizzi, it turns out that TriBITS testing is basically C++ and Python testing. If you can test C++ and Python where CMake is building the C++ code, that is all that is needed for building and running the native TriBITS test suite. I could use some help getting a starting GHA workflow in place with a few different jobs as described above.

I have learned some about GHA in the last month from working on the backend workflow for https://bssw.io and the bssw.io GitHub project.

Speaking of this, Keita mentioned that he would setup a meeting between all of us for mid April to talk about the trilinos work. What if we try to schedule that soon and use to talk about this too and how this fits the current work agreement we have?

Yes, let's coordinate this offline.

[fnrizzi]

@bartlettroscoe if you want, over the next few days I can try to setup a minimal GA workflow file that builds and runs the TriBITS tests to get started with. It would be minimal, but at least something to start from. It should be relatively easy to do since you said testing works like testing in C++. Would that be useful for you? And we can coordinate offline with Keita how to move forward.

[bartlettroscoe]

if you want, over the next few days I can try to setup a minimal GA workflow file that builds and runs the TriBITS tests to get started with

@fnrizzi, that would be great! If the env has C, C++, Fortran, MPI, CMake 3.17, and Python 2.7+, then it should be as easy as:

$ mkdir <build-dir>
$ cd <build-dir>

$ cmake \
  -D TPL_ENABLE_MPI=ON \
  -D CMAKE_C_COMPILER=gcc \
  -D TriBITS_ENABLE_Fortran=ON \
  -D TriBITS_ENABLE_TESTS=ON \
  <tribits-base>

$ make

$ ctest -j16

If you can get that to configure, build, and run the tests, then I can take it from there and get this to submit to CDash and put in a nice link.

I guess you need to develop and test this GA workflow file in your fork of TriBITS (on the 'master' branch)? That seems the only way to test and experiment with GA workflows before going into production.

[fnrizzi]

@bartlettroscoe ok, will try to have this ready by Monday or so, and will ping you then!

[fnrizzi]

@bartlettroscoe
Here is a draft of the yaml fle: https://github.com/fnrizzi/TriBITS/blob/master/.github/workflows/ghactions.yml
I have it running (without MPI for now): https://github.com/fnrizzi/TriBITS/actions/runs/756848014

Some tests fail though:
https://github.com/fnrizzi/TriBITS/runs/2365734531?check_suite_focus=true

I can change the action to run ctest -V to see the full log if you want (which you can also download btw)

[bartlettroscoe]

Here is a draft of the yaml fle

@fnrizzi, very nice! I will take this and and see if can't match the cases listed above. (I think there is 4 build cases to start out with.)

I can change the action to run ctest -V to see the full log if you want (which you can also download btw)

I will set it up to post to CDash so we will be able to see the details very well there (and then I can fix any test failures).

I have a few questions ...

It would be nice to have a single job first run for the configuration:

• CMake 3.17 + Python 2.7 (or 3.x)

and if that passes, then run a set of other configurations like:

• CMake current (3.20) + Python 3.x (or 2.7 if base build is using 3.x)
• CMake 3.17 + Ninja 1.10
• CMake 3.17 + Ninja 1.10 (no Fortran)

That would reduce a lot of wasted computer cycles in case that single build did not work.

Does GitHub Actions support that?

I see this is running with make -j4 and ctest -j4. Doe the VMs running the GitHub action jobs have 4 cores?

[fnrizzi]

@fnrizzi, very nice! I will take this and and see if can't match the cases listed above. (I think there is 4 build cases to start out with.)

@bartlettroscoe Thanks! Do you want me to open a pull request or you just grab the yaml file?

It would be nice to have a single job first run for the configuration:

• CMake 3.17 + Python 2.7 (or 3.x)
  and if that passes, then run a set of other configurations like:
  That would reduce a lot of wasted computer cycles in case that single build did not work.

It is possible to do so, see jobs.<job_id>.needs or Example using status check functions here.
But also note that Jobs run in parallel by default. , so in my view it would make sense to launch the jobs you want. Indeed, If one of the target config lines aborts, all the other jobs are terminated right away. If you are worried about wasting cycles, I don't think this is a big deal, unless you expect to have really heavy jobs, but it seems to me both the build and test suite are pretty quick.

I see this is running with make -j4 and ctest -j4. Doe the VMs running the GitHub action jobs have 4 cores?

Not sure! I actually read once that it is a reasonable assumption, and has always worked fine for me, but I can dig more to see if that really works or if there is a way to automate detecting the parallelism level (I think that is possible to do in Azure for instance)
Here is a link @marcinwrobel1986 informed me of https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners

[bartlettroscoe]

Do you want me to open a pull request or you just grab the yaml file?

@fnrizzi, can you go ahead and clean up your git commits and post a PR? That way you will get credit for this in git.

[fnrizzi]

@bartlettroscoe thanks!
Here it is: #366

[bartlettroscoe]

FYI: I also realized I needed to update the TriBITS GitHub wiki that was still mentioning Travis CI. See here and here.