TriBITS CI testing with GitHub Actions (#363) #389
Conversation
marcinwrobel1986
changed the title
|
@marcinwrobel1986, thanks for putting this together. I can see this ran and submitted to CDash at: Obviously the build names there are all As for the permutation of build jobs, switching up GCC and Fortran compiler versions does not really do as much to test TriBITS as varying other things (but it does not hurt). As stated in #363 (comment) what we really want to toggle the CMake versions, Python versions, Makefile and Ninja generators, Fortran enabled or not enabled, etc. So I think the four builds we want are:
Is that possible to do with this setup? |
bartlettroscoe
changed the title
So we can debug some without spamming CDash
Why not let the jobs run all at the same time?
We have a real GitHub Actions now.
The script now creates and cds into 'tribits-build' subdir.
…ng, improve output (TriBITSPub#363) Hopefully this will run correctly now for all of the build from the GitHub Actions YAML file.
We will need to install docutils if we want to test the generation of documentation.
…TSPub#363) Makes easier to read.
I suspect that some of the tests are not robust enough to work with Ninja as the outer generator. There are some tests that will use Ninja for the inner configure to make sure that it works. Making the entire TriBITS test suite pass using Ninja as the generator in place of Makefiles would be a lot of work. Or, we could conditionally enable many of the tests that depend Unix Makefile logic.
Nice to set just for completeness. Not really related to TriBITSPub#363 but I did this while workoing on that. I also moved -DDART_TESTING_TIMEOUT=60 up in the argument list because -DCTEST_PARALLEL_LEVEL=16 these kind of go together.
This is one of the test cases being enabled for the GitHub Actions builds for TriBITS in TriBITSPub#363. Just reproducing locally and fixing.
…#363) All of these tests fail to configure because we don't pass down the ENABLE_Fortran configure option to the inner CMake configure and if a working Fortran compiler does not even exist on the system, then these tests fail in the configure of the inner project. While I could pass down this info somehow, a lot of these tests would fail anyway without some adjustment. We don't need every test to run if we don't have a Fortran compiler. We just don't want the TriBITS test suite to have a bunch of failing tests if a user clones the repo and tries to build and run the TriBITS test suite on a machine with no Fortran compiler (like a Mac or a Windows machine).
…#314, TriBITSPub#363) With CMake 3.21, with umask=g-rwx,o-rwx you need explicitly set g+rX for the group permissions to be r-X when setting <Project>_MAKE_INSTALL_WORLD_READABLE=ON which was doing just: chmod -R o+rX <dir> Now, when you ony set <Project>_MAKE_INSTALL_WORLD_READABLE=ON, you get: chmod -R g+rX,o+rX <dir> For some reason, with older verisons of CMake, it was setting the default group permission as 'r-x' when it should not have been. I ran exactly the same build on the same machine (crf450) with the same env except I used CMake 3.21.0 vs. 3.17.1 and I had to make this change to get the same permissions on the base install directory. Strange.
For CMake 3.21 it prints a "could not find requested file" instead of "could not find load file". The latter was obviously a typo so it is good they fixed it but I had this checked for in a test. Now the test passes in both cases.
…riBITSPub#363) Seems CMake 3.21 reports two install failures instead of just one as older versions of CMake reported. I guess it depends how you count. Either way, the number is greater than 0 so that is what matters really.
I searched the git logs and it looks like this change was pulled in the CMake 3.18.0 release. But I only tested this with CMake 3.17.1 and 3.21.0.
…ging code (TriBITSPub#363) With CMake 3.21, this was generating a deprecated warning. With the rest of TriBITS requiring CMake 3.17 it seems bad to have code that has min version going back to 2.6. However, this might change the behaivor of this code so that is a warning. (Fingers crossed.)
…ithub-actions-ci (TriBITSPub#363) Merge in the fixes of the tests that should fix all of the nofortran failures and three of the cmake-3.21 failures. But we expect there to still be 10 failing cmake-3.21 failures due to changes in CMake 3.21 behavior vs. CMake 3.17 that appears to be regressions.
|
@marcinwrobel1986, I have just about got it. The only remaining failures are for changes in behavior of CMake 3.21 that I think are likely regressions in CMake 3.21 vs. CMake 3.17. I am going to create a new branch based on this branch 'marcinwrobel1986:377-provide-feedback-on-TriBITS-package-dependencies-data-structures-CI' with the new name '363-github-actions-tests' and a new PR. I will rebase and retain all of the commits (I will just rebase on top of 'master'). (The branch name '377-provide-feedback-on-TriBITS-package-dependencies-data-structures-CI' has the wrong GitHub Issue ID and is is not for the correct issue #363.) On the new branch and PR I will:
(I will add this task list to #363 to complete this.) Some details notes from my latest work on this are given below. DETAILED NOTES: (click to expand). (7/19/2021) Live testing with GitHub Actions is a pain in the rear and is very slow. Therefore, I do as much in a version-controlled script that can be tested locally as I can. I have gotten the drivers working well and now I am just down to failing tests showing the results: showing:
There are 33 test errors for the build As for the build's Three of these tests failed due to changes in behavior of CMake 3.21 vs. CMake 3.17 that seem to be valid changes in behavior. I changed the TriBITS code or tests to adjust for this (and also pass with CMake 3.17). The tests I was able to address are:
(See the commits that impact those tests.) (7/20/2021) I was able to reproduce all of the test falures shown in the GitHub actions builds for the nofortran and cmake-3.21 builds. I have eliminated all of the errors for the no-Fortran builds and I have addressed changes in behavior for CMake 3.21 that appear not to be defects but just changes in behavior (and are possibily improvementments). However, there remains 10 test falures that I reproduced on my RHEL 7 machine crf450 with CMake 3.21 and I posted to CDash are shown at: showing:
These failures bread down into two sets of 5 failing tests. The first set of 5 are those that show runs of ctest -S scripts that should be passing but otherwise return 255 as the return code shown in: showing:
These are all cases where the ctest -S driver reports all passing and there is no hint in the output of anything failing but yet have the return code 255. The second set of 5 failing tests are shown in: showing:
These are all cases that have build errors. It looks like CMake 3.21 is refusing to generate tests when there is a build error where CMake 3.17 still generated tests even when there was a build error. That is not good because we don't want to take down all of our testing just because one test executable does not build. I will need to report these errors. |
Brings in numerous refactorings to TriBITS over the last 3 months, but there should be no breaks in backward compatibility. Almost every file in TriBITS is changed due to the lower-casing of command, macro and function names in PR TriBITSPub/TriBITS#379. But the main driver for this snapshot is to bring in the change in PR TriBITSPub/TriBITS#413 that should make it so that Kokkos INTERFACE_COMPILE_OPTIONS get propagated to downstream targets in TriBITS and therefore to external customers through installed <Package>Config.cmake files and IMPORTED targets. I should have done several snapshots in the last few months and not done a big snapshot like this (but I have been testing with Trilinos locally along the way). Overall, this merge brings in changes from a bunch of TriBITS PRs including (from most recent): * TriBITSPub/TriBITS#413: Change internal TriBITS target_link_libraries() to PUBLIC (TriBITSPub/TriBITS#299) component: core type: enhancement * TriBITSPub/TriBITS#410: Upgrade from cmake 3.21.0 to 3.21.2 (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) * TriBITSPub/TriBITS#394: DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#409: Add getTestDictStatusField() to handle empty 'status' field (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#408: Deal with spaces in CDash url parts (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#403: Spelling fixes * TriBITSPub/TriBITS#407: Move tribits_get_build_url_and_write_to_file() to stand-alone module (TriBITSPub/TriBITS#154) component: ctest_driver type: enhancement * TriBITSPub/TriBITS#388: Fixing typos (TriBITSPub/TriBITS#377) * TriBITSPub/TriBITS#406: Fix tribits_ctest_driver() package-by-package mode for CMake 3.19+ (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) component: ctest_driver type: bug * TriBITSPub/TriBITS#401: Improve GitHub Actions and CDash integration (TriBITSPub/TriBITS#154) type: enhancement * TriBITSPub/TriBITS#366: CI: draft action yaml * TriBITSPub/TriBITS#402: Revert some incorrect uppercase->lowercase changes * TriBITSPub/TriBITS#387: Build and deploy TriBITS documentation with Sphinx (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#398: Deal with pr null in not preprending build name (TriBITSPub/TriBITS#363) type: bug * TriBITSPub/TriBITS#396: Send PR results to 'Pull Request' CDash group and add PR ID to build names (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#397: Print the ninja path and version (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#393: GitHub Actions based testing for TriBITS (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#389: TriBITS CI testing with GitHub Actions (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#392: Fix broken tests for non-Fortran and CMake 3.21 builds (TriBITSPub/TriBITS#363) component: core * TriBITSPub/TriBITS#391: Fix up build_docs.sh for Sphinx doc generation (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#390: Add test for doc generation and fix usage of Python3 component: documentation type: bug * TriBITSPub/TriBITS#385: Replace last few references to TribitsDevelopersGuide.html (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#384: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#383: Remove endfunction(<string>) and endmacro(<string>) (TriBITSPub/TriBITS#274, TriBITSPub/TriBITS#382) component: common_tpls type: bug * TriBITSPub/TriBITS#380: More package-arch data-structure documentation updates (TriBITSPub/TriBITS#63) component: documentation type: enhancement * TriBITSPub/TriBITS#379: Convert TriBITS to lower-case CMake command, macro, and function names (TriBITSPub/TriBITS#274) The following files were conflicting where I went with what is on the Trilinos 'develop' branch: * cmake/tribits/common_tpls/FindTPLBLAS.cmake * cmake/tribits/common_tpls/FindTPLLAPACK.cmake * cmake/tribits/common_tpls/FindTPLNetcdf.cmake (It looks like the above changes never made it back into TriBITS proper. The conflicts were due to the case changes in cmake command calls in these files due to TriBITSPub/TriBITS#379.) There was also a conflict in the file: * cmake/tribits/core/installation/TribitsProjectConfigTemplate.cmake.in I looked at that one carefully and I think that may have been due to fixes on both sides and then the case changes from TriBITSPub/TriBITS#379.
Implements #363
This PR:
provides tribits testing as it was defined with
.travis.ymlfiletesting with 4 different combinations:
@bartlettroscoe please take a look if that's what you wanted, thanks!
Tasks:
CTEST_BUILD_NAMEthat includes CMake version, generator and version (i.e. makefiles or ninja), compiler and version, no fortran (@bartlettroscoe)