Ci rework gcc builds #3631

scottwittenburg · 2023-05-24T16:16:31Z

This PR reworks how docker images are built for gcc, clang, and intel builds done in github actions. To dramatically reduce build times for the gcc builds, the plan going forward is to use pre-built Docker images from the E4S project (see "Minimal Images", described here) as a base image. These images come with some spack packages already installed, but more importantly, installing ADIOS2 dependencies using the spack shipped in those images results in most dependencies getting installed from a release-specific E4S buildcache. This is far faster than building those dependencies from sources.

For intel builds run in el8, since no buildcache is available to install ADIOS2 dependencies for that OS, the dependencies that can be installed from the package manager are installed, and options for missing dependencies are just turned off.

After this PR is merged, there are only two shell scripts used to build, tag, and push all the images used in gcc and intel builds on github actions. To build gcc images (currently including gcc8, gcc9, gcc10, gcc11, and clang10), simply run scripts/ci/images-v2/build-ubuntu.sh. On a fairly old linux machine with 32 hyperthreads, but with slow spinning disks, and with no caching, this now takes around 30 minutes. To build, tag, and push images for icc and oneapi testing, run scripts/ci/images-v2/build-el8.sh. This takes around 20 minutes, under the same conditions mentioned above.

This PR also renders some of the existing dockerfiles, scripts, cmake files, etc unused, and thus removes them from the repo.

fixes: #3611
fixes: #3612
fixes: #3613
fixes: #3614

scottwittenburg · 2023-06-23T18:51:37Z

@vicentebolea The two failing tests on ubuntu/gcc10/mpich seem to have failed on master recently as well.

There is also 1 configure warning on each of the four intel builds. I think it's because those images are using the latest nightly version of cmake, which is warning about kwsys having too old of a minimum cmake version requirement.

Other than those issues, this is probably ready for you to review.

vicentebolea

Review from the skies, I will review the rest tomorrow, running out of battery in the fly.

scripts/ci/images-v2/Dockerfile.ci-spack-ubuntu20.04-leaf

scripts/ci/setup-run/ci-ubuntu20.04-gcc10.sh

.shellcheck_exclude_paths

.github/workflows/everything.yml

scripts/ci/cmake-v2/ci-ubuntu20.04-clang10-mpi.cmake

scottwittenburg · 2023-06-30T21:43:26Z

@vicentebolea Since a lot of the work that went into this PR was exploratory at first, the commit history is far from clean. I'm not sure it's worth the effort to try to break it up into logical chunks now, but I could easily collapse it into a single commit if you want.

vicentebolea

Great job Scott, only cosmetic changes requested here. Sorry for me taking this long to review this PR, the reason is due to the size of changes and me being distracted with the AHM this week. Let me know

scripts/ci/cmake-v2/ci-el8-icc-mpi.cmake

scripts/ci/cmake-v2/ci-el8-icc-serial.cmake

scripts/ci/images-v2/Dockerfile.ci-spack-ubuntu20.04-leaf

scripts/ci/cmake-v2/ci-ubuntu20.04-gcc10-mpich.cmake

vicentebolea · 2023-06-30T21:45:47Z

scripts/ci/cmake-v2/ci-ubuntu20.04-gcc11-mpi.cmake

@@ -28,6 +35,7 @@ MPIEXEC_EXTRA_FLAGS:STRING=--allow-run-as-root --oversubscribe
 MPIEXEC_MAX_NUMPROCS:STRING=${N2CPUS}
 ")

-set(CTEST_CMAKE_GENERATOR "Ninja")
+set(CTEST_TEST_ARGS PARALLEL_LEVEL 1)
+set(CTEST_CMAKE_GENERATOR "Unix Makefiles")


Same as above

scripts/ci/setup-run/ci-ubuntu20.04-gcc10.sh

scripts/ci/setup-run/ci-ubuntu20.04-gcc11.sh

scripts/ci/setup-run/ci-ubuntu20.04-gcc8.sh

scripts/ci/setup-run/ci-ubuntu20.04-gcc9.sh

source/adios2/engine/bp5/BP5Reader.cpp

vicentebolea · 2023-06-30T21:58:01Z

We also are missing the clang6 build but we can provide it in a later PR

vicentebolea · 2023-06-30T23:08:45Z

I'm not sure it's worth the effort to try to break it up into logical chunks now, but I could easily collapse it into a single commit if you want.

Its up to you, but due to the large amount of changes you might as well make a single commit and be done with it

scottwittenburg · 2023-07-07T19:53:06Z

You mentioned you tried using the slave approach and it worked for you:

I Tried update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11 in the ubuntu image and it seems to work.

Here's what I see:

$ docker run --rm -ti ecpe4s/ubuntu20.04:23.02
root@9bfae9858561:/# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [37.5 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,529 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,346 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,866 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,070 kB]   
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,671 kB]    
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,369 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [40.3 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 14.3 MB in 2s (6,591 kB/s)                           
Reading package lists... Done
root@9bfae9858561:/# update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11
update-alternatives: error: alternative g++ can't be slave of gcc: it is a master alternative

Let me know if it's obvious to you what I'm doing wrong here, but I just copy-pasted what you said worked for you.

scottwittenburg · 2023-07-07T20:04:00Z

I'm not sure it's worth the effort to try to break it up into logical chunks now, but I could easily collapse it into a single commit if you want.

Its up to you, but due to the large amount of changes you might as well make a single commit and be done with it

Yeah ok, once you're happy with the changes (and everything is still working), I'll squash to a single commit and push one last time.

scottwittenburg · 2023-07-07T21:16:58Z

5b4dc07 adds a clang6 build, since it was easy to add, so we'll see how it fares in CI. But if you prefer I can also save that commit for a follow-on PR.

vicentebolea · 2023-07-07T21:17:49Z

Let me know if it's obvious to you what I'm doing wrong here, but I just copy-pasted what you said worked for you.

I see, it appears that g++ is at the same time another master alternative in this E4s image, thus update-alternatives is declining the instruction. In vanilla ubuntu20.04 is possible but not in this image, we tried but it is all right is not super important. Lets move on with this

vicentebolea

Looks great, before squashing everything into a single commit add a commit with the changes for source/adios2/engine/bp5/BP5Reader.cpp.

scripts/ci/cmake-v2/ci-ubuntu20.04-gcc11-mpi.cmake

vicentebolea · 2023-07-07T21:52:57Z

scripts/ci/cmake-v2/ci-ubuntu20.04-gcc8-mpi.cmake

@@ -28,6 +35,7 @@ MPIEXEC_EXTRA_FLAGS:STRING=--allow-run-as-root --oversubscribe
 MPIEXEC_MAX_NUMPROCS:STRING=${N2CPUS}
 ")

+set(CTEST_TEST_ARGS PARALLEL_LEVEL 1)


scripts/ci/cmake-v2/ci-ubuntu20.04-gcc9-mpi.cmake

scottwittenburg · 2023-07-07T22:06:54Z

before squashing everything into a single commit add a commit with the changes for source/adios2/engine/bp5/BP5Reader.cpp

Should I separate out the change to DataManWriter.cpp in that same commit as well?

vicentebolea · 2023-07-07T22:13:41Z

before squashing everything into a single commit add a commit with the changes for source/adios2/engine/bp5/BP5Reader.cpp

Should I separate out the change to DataManWriter.cpp in that same commit as well?

Yep

vicentebolea

The two failing test for mpich are not related to this PR.

vicentebolea

followup from previous review; everything else good; lets try ghcr instead of docker.

scripts/ci/images-v2/Dockerfile.ci-spack-ubuntu20.04-base

scottwittenburg · 2023-07-11T16:07:59Z

@vicentebolea Do you want to take one last look before merging? I think I've addressed everything we discussed.

vicentebolea

Lookgs good small last detail

vicentebolea · 2023-07-11T16:20:01Z

scripts/ci/images-v2/build-ubuntu.sh

+docker build --rm --build-arg CLANG_VERSION=6.0 -f ./Dockerfile.ci-spack-ubuntu20.04-clang -t adios2:ci-spack-ubuntu20.04-clang6 .
+docker build --rm --build-arg CLANG_VERSION=10 -f ./Dockerfile.ci-spack-ubuntu20.04-clang -t adios2:ci-spack-ubuntu20.04-clang10 .
+
+# Tag test images for pushing (TODO: change organization before merge)


Suggested change

# Tag test images for pushing (TODO: change organization before merge)

No need as we will use GHCR

vicentebolea

Ready to merge; If you cant merge let me know.

Good job there :)

scottwittenburg · 2023-07-11T20:02:25Z

Thanks @vicentebolea! It seems that I cannot merge, but I don't think it's about permissions, as it seems I could push the "Enable auto-merge" button to "Automatically merge when all requirements are met". Is it possible the failing mpich test is preventing merge?

vicentebolea · 2023-07-11T20:04:20Z

@scottwittenburg yeah, I have also rebase it, lets merge it after passing CI

…rk-gcc-builds

scottwittenburg requested a review from vicentebolea as a code owner May 24, 2023 16:16

scottwittenburg force-pushed the ci-rework-gcc-builds branch 5 times, most recently from fe48235 to 16fcd4d Compare May 24, 2023 19:58

scottwittenburg force-pushed the ci-rework-gcc-builds branch 2 times, most recently from bdfaebf to 0a7569a Compare June 20, 2023 22:24

vicentebolea requested changes Jun 29, 2023

View reviewed changes

scottwittenburg force-pushed the ci-rework-gcc-builds branch from 83a86cc to aa1ad76 Compare June 30, 2023 21:26

scottwittenburg requested a review from vicentebolea June 30, 2023 21:43

vicentebolea requested changes Jun 30, 2023

View reviewed changes

vicentebolea requested changes Jul 7, 2023

View reviewed changes

scottwittenburg force-pushed the ci-rework-gcc-builds branch from 5b4dc07 to ec825fe Compare July 7, 2023 23:06

vicentebolea approved these changes Jul 10, 2023

View reviewed changes

vicentebolea requested changes Jul 10, 2023

View reviewed changes

scripts/ci/images-v2/Dockerfile.ci-spack-ubuntu20.04-base Outdated Show resolved Hide resolved

scottwittenburg force-pushed the ci-rework-gcc-builds branch 4 times, most recently from 7b7e8b1 to e911aa7 Compare July 10, 2023 23:12

vicentebolea requested changes Jul 11, 2023

View reviewed changes

scottwittenburg force-pushed the ci-rework-gcc-builds branch from e911aa7 to fa274a9 Compare July 11, 2023 16:37

vicentebolea mentioned this pull request Jul 11, 2023

Add Libcurl in some of our Github builds #3694

Closed

vicentebolea approved these changes Jul 11, 2023

View reviewed changes

scottwittenburg added 2 commits July 11, 2023 16:03

ci: Rework gcc/clang/intel builds

008b516

Fix warnings newly detected by latest oneapi

74c5b9d

vicentebolea force-pushed the ci-rework-gcc-builds branch from fa274a9 to 74c5b9d Compare July 11, 2023 20:03

vicentebolea added the backport:release label Jul 11, 2023

vicentebolea merged commit 9965921 into ornladios:master Jul 11, 2023
28 of 29 checks passed

scottwittenburg deleted the ci-rework-gcc-builds branch July 11, 2023 21:49

vicentebolea added a commit to vicentebolea/ADIOS2 that referenced this pull request Jul 12, 2023

fixup! Merge pull request ornladios#3631 from scottwittenburg/ci-rewo…

ff0ec46

…rk-gcc-builds

vicentebolea added a commit to vicentebolea/ADIOS2 that referenced this pull request Jul 12, 2023

fixup! Merge pull request ornladios#3631 from scottwittenburg/ci-rewo…

3a1ae8d

…rk-gcc-builds

vicentebolea removed the backport:release label Aug 2, 2023

BrewTestBot mentioned this pull request Apr 4, 2024

adios2 2.10.0 Homebrew/homebrew-core#167957

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ci rework gcc builds #3631

Ci rework gcc builds #3631

scottwittenburg commented May 24, 2023 •

edited by vicentebolea

scottwittenburg commented Jun 23, 2023

vicentebolea left a comment

scottwittenburg commented Jun 30, 2023

vicentebolea left a comment

vicentebolea Jun 30, 2023

vicentebolea commented Jun 30, 2023

vicentebolea commented Jun 30, 2023

scottwittenburg commented Jul 7, 2023

scottwittenburg commented Jul 7, 2023

scottwittenburg commented Jul 7, 2023

vicentebolea commented Jul 7, 2023

vicentebolea left a comment

vicentebolea Jul 7, 2023

scottwittenburg commented Jul 7, 2023

vicentebolea commented Jul 7, 2023

vicentebolea left a comment

vicentebolea left a comment

scottwittenburg commented Jul 11, 2023

vicentebolea left a comment

vicentebolea Jul 11, 2023

vicentebolea left a comment

scottwittenburg commented Jul 11, 2023

vicentebolea commented Jul 11, 2023

Ci rework gcc builds #3631

Ci rework gcc builds #3631

Conversation

scottwittenburg commented May 24, 2023 • edited by vicentebolea

scottwittenburg commented Jun 23, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

scottwittenburg commented Jun 30, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

vicentebolea Jun 30, 2023

Choose a reason for hiding this comment

vicentebolea commented Jun 30, 2023

vicentebolea commented Jun 30, 2023

scottwittenburg commented Jul 7, 2023

scottwittenburg commented Jul 7, 2023

scottwittenburg commented Jul 7, 2023

vicentebolea commented Jul 7, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

vicentebolea Jul 7, 2023

Choose a reason for hiding this comment

scottwittenburg commented Jul 7, 2023

vicentebolea commented Jul 7, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

vicentebolea left a comment

Choose a reason for hiding this comment

scottwittenburg commented Jul 11, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

vicentebolea Jul 11, 2023

Choose a reason for hiding this comment

vicentebolea left a comment

Choose a reason for hiding this comment

scottwittenburg commented Jul 11, 2023

vicentebolea commented Jul 11, 2023

scottwittenburg commented May 24, 2023 •

edited by vicentebolea