single dockerfile and single step in the docker building CI #863

shimwell · 2023-03-02T16:11:41Z

Description

This PR is an alternative to #822
In addition to moving the scripts into a single dockerfile this PR also changes the CI for publishing dockerfiles.
Currently we had a branching and caching approach to the CI dockerfile building that is efficient but complex.
This approach reduces the complexity of the CI and allows all the perturbations to run in parallel.

This might actually end up being quicker to build and quicker to fail when something is broken as the CI doesn't have breaks waiting for all the docker stages to reach the same stage to proceed to the next stage. Also if caching can be figured out then this workflow could make use of pre-built images to be very fast

Motivation and Context

simpler is nice for maintainers 😄

Changes

refactored CI

Behavior

single dockerfile instead of scripts and simpler CI

shimwell · 2023-03-02T16:15:37Z

Action is running on my branch with 24 jobs in parallel
https://github.com/shimwell/DAGMC/actions/runs/4315972934

shimwell · 2023-03-02T20:13:42Z

All jobs on my fork CI passed, it took 3 hours 28mins

gonuke

Thanks for getting this started @shimwell.

I think it went a little to far/too fast and has lost the narrative of what we want to do:

build docker images to be used by CI later on that only have appropriate dependencies - up to moab (this part can proceed in parallel)
push those images to the registry to be used later using a temporary tag
build and test DAGMC using those images as the container (this part can proceed in parallel)
rename the docker images in the registry to a more stable tag

BONUS: figure out how to populate a cache for the first docker build using the existing docker images available in the registry. (separate PR?)

gonuke · 2023-03-03T11:30:42Z

.github/workflows/docker_publish.yml

      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
+        uses: docker/setup-qemu-action@v2


We don't need QEMU

QEMU has been removed from the yml file so I think this is resolved

gonuke · 2023-03-03T11:31:52Z

.github/workflows/docker_publish.yml

        with:
          file: CI/Dockerfile
-          target: external_deps
+          target: dagmc


Recall that this should only build the images to be used in CI for building/testing DAGMC.

We should only build to moab, and then test using this image for building DAGMC as we will later be doing in CI.

server-stage of multi stage build is set to moab so I think this is now resolved

gonuke · 2023-03-03T11:33:53Z

.github/workflows/docker_publish.yml

-        uses: actions/checkout@v3
+            MOAB=${{ matrix.moab_versions }}
+            build_mw_reg_tests=ON
+          tags: ghcr.io/${{ github.repository_owner }}/dagmc-ci-ubuntu-${{ matrix.ubuntu_versions }}-${{ matrix.compiler}}-ext-hdf5_${{ matrix.hdf5_versions}}-moab_${{ matrix.moab_versions }}-dagmc:ci_testing


If this is push: false then tags don't matter, but I think we should push.

pushing now taken care of by the multistage docker build action so I think this is resolved

gonuke · 2023-03-03T11:34:42Z

.github/workflows/docker_publish.yml

      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
+        if: ${{ github.repository_owner == 'svalinn' }}
+        uses: docker/setup-qemu-action@v2


Also don't need QEMU here

QEMU has been removed from the yml file so I think this is resolved

gonuke · 2023-03-03T15:17:42Z

.github/workflows/docker_publish.yml

-
-  build-housekeeping-img:
-    needs: build-base-img
+  build-test-dagmc-img:


rename based on comments below

this has now been renamed to build-dependency-img so I think this is resolved

.github/workflows/docker_publish.yml

shimwell · 2023-03-03T18:29:58Z

Had a go at all those comments and pushed, CI is running on my fork https://github.com/shimwell/DAGMC/actions/runs/4325712183

shimwell · 2023-03-03T23:34:47Z

CI on fork passed https://github.com/shimwell/DAGMC/actions/runs/4325712183

2nd stage only took a few seconds so the local stage cache appears to have worked.

This used the local stage cache instead of the container repo. But the layer has the same tag locally and I'm the container repo.

Perhaps this is not what we want

gonuke

This definitely looks simpler and probably easier to maintain over the long term. I think there is still a discussion point about how to use the docker image cache.

gonuke · 2023-03-04T17:20:43Z

.github/workflows/docker_publish.yml

+         OFF,
+         ON,


We don't appear to be passing this into the docker build or the tag naming and therefore have duplicate images being built and tagged. We should discuss whether we really need the statically linked version. Not sure I remember who requires/desires that.

static_exe has been removed from the matrix so I think this is resolved

gonuke · 2023-03-04T17:21:27Z

.github/workflows/docker_publish.yml

-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
+            build_mw_reg_tests=ON
+            dependency_image_location=ghcr.io/${{ github.repository_owner }}/dagmc-ci-ubuntu-${{ matrix.ubuntu_versions }}-${{ matrix.compiler}}-ext-hdf5_${{ matrix.hdf5_versions}}-moab_${{ matrix.moab_versions }}:ci_testing


happy to see this working, but wonder if we are better off with a different strategy that uses the cache instead? something to discuss.

dependency_image_location has now been removed and the PR has moved to a multistage build so I think this is resolved

gonuke

a couple of additional comments

gonuke · 2023-03-09T02:15:33Z

.github/workflows/docker_publish.yml

@@ -141,14 +63,17 @@ jobs:
            UBUNTU_VERSION=${{ matrix.ubuntu_versions }}
            COMPILER=${{ matrix.compiler }}
            HDF5=${{ matrix.hdf5_versions }}
-          tags: ghcr.io/${{ github.repository_owner }}/dagmc-ci-ubuntu-${{ matrix.ubuntu_versions }}-${{ matrix.compiler}}-ext-hdf5_${{ matrix.hdf5_versions}}:ci_testing
+            MOAB=${{ matrix.moab_versions }}
+            build_mw_reg_tests=ON


don't we want this off?

this build arg has been removed from the yml and dockerfile so I think this is now resolved

gonuke · 2023-03-09T02:16:54Z

.github/workflows/docker_publish.yml

-    steps:
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
+            build_mw_reg_tests=ON


Don't we want this off?

this build arg has been removed from the yml and dockerfile so I think this is now resolved

pshriwise · 2023-03-10T19:31:06Z

.github/workflows/docker_publish.yml

        with:
          file: CI/Dockerfile
-          target: moab
+          target: dagmc_test
          context: .


I'm thinking we may not need this line in this case. The action documentation indicates that the right version of the repository will be checked out in the Docker image by default. This isn't true in the case of our dependencies though, so I can see why we'd need it there.

this line has been removed from the yml file so I think this one is resolved

… tests

bquan0 · 2023-04-20T01:00:00Z

I just opened a PR to the branch this PR is on to replace the docker build actions with the multistage build action.

Use multistage docker build action

gonuke · 2023-04-24T22:54:03Z

@shimwell - this appears to need a rebase...

gonuke · 2023-04-24T22:57:20Z

This also failed here (https://github.com/shimwell/DAGMC/actions/runs/4752350398/jobs/8442595961) - apparently it failed to push to GHCR? and it was successful for @bquan0 here (https://github.com/bquan0/DAGMC/actions/runs/4749240986/jobs/8436860572) but was based on cached version??

bquan0 · 2023-04-25T17:46:40Z

I ran into that problem a few times on my workflow runs too and I mentioned it in the PR. I solved it by going to the settings of the package it was trying to push to, then checking the DAGMC repo under the "Manage Actions access" section.

gonuke · 2023-04-30T19:21:38Z

Another PR for some final cleanup

gonuke · 2023-04-30T20:10:57Z

Another PR for some final cleanup

Github actions for this PR are successful

Two small changes to clean things up a little further

shimwell · 2023-04-30T21:27:58Z

Thanks Paul, I've merged that in.

gonuke · 2023-04-30T22:16:33Z

Confirmation of success here

gonuke

I think this is in good shape and is a useful model for many projects going forward. I'd love another review since I had a hand in authoring this in the end

pshriwise

Some small line comments from me, but this looks really nice! I was expecting a much larger changeset to accomplish this.

Q: Which of the stages gets uploaded in the end?

.github/workflows/docker_publish.yml

pshriwise · 2023-05-04T16:25:08Z

.github/workflows/docker_publish.yml

Is the bk0 suffix specific to something or arbitrary?

not sure, but I see bk0 appears in two places. perhaps @gonuke or @bquan0 know where that suffix came from

its generated by the multistage-build-action - not sure why the author uses that string.

CI/Dockerfile

.github/workflows/docker_publish.yml

Co-authored-by: Patrick Shriwise <pshriwise@gmail.com>

gonuke · 2023-05-04T21:21:31Z

Q: Which of the stages gets uploaded in the end?

Every stage that is explicitly referenced in a multistage-docker-build-action is pushed to the repo:

initially: base, external_deps, hdf5, moab
during testing: dagmc, dagmc_test

They each get pushed with the custom tag and then we convert push the dagmc stage with tags stable and latest as well, but only when running from the svalinn repo.

Co-authored-by: Paul Wilson <paul.wilson@wisc.edu>

pshriwise · 2023-05-05T21:05:13Z

I'd love to approve and merge this, but there a runner isn't picking up the Mac testing job unfortunately.

gonuke · 2023-05-05T23:11:39Z

Saw that...☹️

gonuke · 2023-05-06T22:57:58Z

Github has turned off the MacOS 10.15 runners, so I made this PR to move us forward.

update MacOS runner

gonuke · 2023-05-07T10:23:32Z

This is still passing here: https://github.com/shimwell/DAGMC/actions/runs/4904019862

pshriwise

Looking forward to utilizing these improvements in future PRs!! Thanks @gonuke and @shimwell for all the detective work 🤩 📈

gonuke · 2023-05-07T15:34:07Z

Thanks for the teamwork @shimwell @bquan0 & @pshriwise !

shimwell · 2023-05-07T16:28:37Z

Delighted this one got in, a real team effort. Nice work all. What shall we do next 😁

pshriwise · 2023-05-07T18:38:53Z

😢 https://github.com/svalinn/DAGMC/actions/runs/4907989513/jobs/8764666815

shimwell · 2023-05-07T19:33:07Z

posting error message here so we don't lose it when the CI gets old

[ 55%] Building CXX object test/CMakeFiles/mbcn_test.dir/mbcn_test.cpp.o
Installing collected packages: pymoab
  Found existing installation: pymoab 5.4.1
    Can't uninstall 'pymoab'. No files were found to uninstall.
  Running setup.py develop for pymoab
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab:
    running develop
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 2] No such file or directory: '/root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages/test-easy-install-5760.write-test'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages
    
    This directory does not currently exist.  Please create it and try again, or
    choose a different installation directory (using the -d or --install-dir
    option).
    
    
    ----------------------------------------
  Can't roll back pymoab; was not uninstalled
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab" failed with error code 1 in /root/build_dir/moab/bld/pymoab/
pymoab/CMakeFiles/pymoab-local-install.dir/build.make:58: recipe for target 'pymoab/CMakeFiles/pymoab-local-install' failed
make[2]: *** [pymoab/CMakeFiles/pymoab-local-install] Error 1
CMakeFiles/Makefile2:1238: recipe for target 'pymoab/CMakeFiles/pymoab-local-install.dir/all' failed
make[1]: *** [pymoab/CMakeFiles/pymoab-local-install.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 55%] Linking CXX executable ../bin/mbcn_test
[ 55%] Built target mbcn_test
make: *** [all] Error 2
Makefile:140: recipe for target 'all' failed
Error: Process completed with exit code 2.

shimwell · 2023-05-07T19:42:17Z

e08954f49424: Pull complete
Digest: sha256:3c332407c190c017dbe049e8e9c9d54c5e44166b484c56fd414821bc54e5c233
Status: Downloaded newer image for akhilerm/repo-copy:latest
crane [copy ghcr.io/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0/dagmc:refs_heads_develop-bk0 ghcr.io/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0:stable]
2023/05/07 17:46:34 Copying from ghcr.io/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0/dagmc:refs_heads_develop-bk0 to ghcr.io/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0:stable
Error: fetching "ghcr.io/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0/dagmc:refs_heads_develop-bk0": GET https://ghcr.io/v2/svalinn/dagmc-ci-ubuntu-22.04-gcc-ext-hdf5_1.10.4-moab_5.3.0/dagmc/manifests/refs_heads_develop-bk0: MANIFEST_UNKNOWN: manifest unknown
panic: exit status 1

goroutine 1 [running]:
main.main()
	/go/src/github.com/akhilerm/repo-copy/main.go:20 +0x15a
Error: The process '/usr/bin/docker' failed with exit code 2

gonuke · 2023-05-08T01:18:28Z

Yep - working on that, too ☹️

shimwell added 4 commits March 2, 2023 16:05

single ci step

038a334

added change log

46b88ea

removed needs part of step

267a525

Merge branch 'develop' into single_dockerfile_parallel_ci

fe20f3d

gonuke requested changes Mar 3, 2023

View reviewed changes

shimwell added 2 commits March 3, 2023 17:12

added option of dagmc layer built on any base

b2283f5

two steps with first base image push

0087c68

gonuke requested changes Mar 4, 2023

View reviewed changes

gonuke requested changes Mar 9, 2023

View reviewed changes

pshriwise reviewed Mar 10, 2023

View reviewed changes

gonuke mentioned this pull request Mar 16, 2023

move test-on-merge with master/develop to be optional #870

Merged

bquan0 added 3 commits April 11, 2023 16:58

replace docker builds with multistage action

17f4a13

updated image names of Push Image jobs, comment out if statements for…

24f5851

… tests

clean up docker_publish.yml

cb8fa44

bquan0 mentioned this pull request Apr 20, 2023

Use multistage docker build action shimwell/DAGMC#5

Merged

Merge pull request #5 from bquan0/multistage-action

6ef0281

Use multistage docker build action

Merge branch 'develop' into single_dockerfile_parallel_ci

d8ee7a1

shimwell added 5 commits April 27, 2023 06:51

removed dependency_image_location from docker

a2c7aef

removed build_mw_reg_tests which is set to off in dockerfile

7d5a0ed

removed unused ARG build_mw_reg_tests as

36aabf6

[skip ci] removed commented static dagmc build

e912134

removed unused static_exe from matrix

4b39421

Merge pull request #7 from gonuke/single_dockerfile_parallel_ci

e4740dc

Two small changes to clean things up a little further

gonuke mentioned this pull request May 1, 2023

Use multistage docker build action in docker_publish.yml #875

Closed

gonuke approved these changes May 1, 2023

View reviewed changes

gonuke requested a review from pshriwise May 4, 2023 13:06

pshriwise requested changes May 4, 2023

View reviewed changes

shimwell commented May 4, 2023

View reviewed changes

.github/workflows/docker_publish.yml Outdated Show resolved Hide resolved

shimwell commented May 4, 2023

View reviewed changes

.github/workflows/docker_publish.yml Outdated Show resolved Hide resolved

Apply suggestions from code review from @pshriwise

2e52383

Co-authored-by: Patrick Shriwise <pshriwise@gmail.com>

adding build type release to moab cmake

224259e

Co-authored-by: Paul Wilson <paul.wilson@wisc.edu>

update MacOS runner

e77ccb2

Merge pull request #8 from gonuke/single_dockerfile_parallel_ci

190910c

update MacOS runner

pshriwise approved these changes May 7, 2023

View reviewed changes

pshriwise merged commit 8af1efe into svalinn:develop May 7, 2023

gonuke mentioned this pull request May 7, 2023

Simplify testing strategy gonuke/DAGMC#4

Merged

gonuke mentioned this pull request May 8, 2023

Streamline and simplify CI building #880

Merged

single dockerfile and single step in the docker building CI #863

single dockerfile and single step in the docker building CI #863

Conversation

shimwell commented Mar 2, 2023

Description

Motivation and Context

Changes

Behavior

shimwell commented Mar 2, 2023

shimwell commented Mar 2, 2023

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shimwell commented Mar 3, 2023

shimwell commented Mar 3, 2023 • edited Loading

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shimwell Apr 27, 2023 • edited Loading

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bquan0 commented Apr 20, 2023

gonuke commented Apr 24, 2023

gonuke commented Apr 24, 2023 • edited Loading

bquan0 commented Apr 25, 2023

gonuke commented Apr 30, 2023

gonuke commented Apr 30, 2023

shimwell commented Apr 30, 2023

gonuke commented Apr 30, 2023

gonuke left a comment

Choose a reason for hiding this comment

pshriwise left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gonuke May 4, 2023 • edited Loading

Choose a reason for hiding this comment

gonuke commented May 4, 2023

pshriwise commented May 5, 2023

gonuke commented May 5, 2023

gonuke commented May 6, 2023

gonuke commented May 7, 2023

pshriwise left a comment

Choose a reason for hiding this comment

gonuke commented May 7, 2023

shimwell commented May 7, 2023 • edited Loading

pshriwise commented May 7, 2023

shimwell commented May 7, 2023

shimwell commented May 7, 2023

gonuke commented May 8, 2023

shimwell commented Mar 3, 2023 •

edited

Loading

shimwell Apr 27, 2023 •

edited

Loading

gonuke commented Apr 24, 2023 •

edited

Loading

gonuke May 4, 2023 •

edited

Loading

shimwell commented May 7, 2023 •

edited

Loading