Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GEOS-Chem CI Pipelines! #112

Merged
merged 2 commits into from Oct 25, 2019
Merged

GEOS-Chem CI Pipelines! #112

merged 2 commits into from Oct 25, 2019

Conversation

LiamBindle
Copy link
Contributor

@LiamBindle LiamBindle commented Oct 24, 2019

GEOS-Chem CI Pipelines!

Hi everyone,

This is a pull request for adding CI pipelines to GEOS-Chem Classic. I've set these CI pipelines up on my fork, and I've given you all the permissions to commit/push/tag/release on my fork so that you can try it out if you want. You can find the Azure pipelines for my GEOS-Chem fork here: https://dev.azure.com/lrbindle/geos-chem/_build?definitionId=10 (let me know if you can't access that link).

Sorry, this PR is long because there are a number of moving parts. Please don't think this means any cause for concern though, because the Azure pipeline is independent of the GitHub repo (besides a few YAML config files). I think we can start moving forward with this any time now.

I set up three pipelines. The first, "quick-build", is a fast and simple build pipeline that tries to compile GEOS-Chem in a "normal" build environment. This pipeline is triggered by commits to dev/* and bugfix/*, as well as pull requests (and commits to open pull requests). The second, "build-matrix", tries to build GEOS-Chem in 11 different environments (I call them lines) which each focus on a specific GEOS-Chem dependency (e.g. there are lines for GCC 4-9, lines for old and new NetCDF's, etc.). This pipeline is triggered by pre-releases and commits to the master branch. The third pipeline, "release", builds a docker container with GEOS-Chem prebuilt in it (with separate executables for standard, tropchem, and soa_svpoa mechanisms) and automatically pushes it to Docker Hub. This pipeline is triggered by GitHub releases.

Note: The build matrix pipeline already identified a problem with building with GCC 9! (see #113)

Please feel free to play around with pushing, making PRs, merging, and tagging on my GEOS-Chem fork to explore the behavior if you're interested (there's nothing important on my fork...I plan to re-fork soon anyways). Note that I've added some weird tags like 12.7.4 to my fork to test the CI behaviour...ignore these.

Moving forward

To merge this PR a few things need to happen. These things will need to be done by someone with admin privileges for the GEOS-Chem repo. The first of these is setting up the Azure pipelines. To do this, they need to:

  1. Create a GEOS-Chem organization account on Azure DevOps (I think you can do that here) (try to do this with the GEOS-Chem GitHub organization if it's possible)
  2. Create a GEOS-Chem project (GEOS-Chem Classic and GCHP should have their own projects)
  3. Click on the GEOS-Chem project
  4. Click on "Pipelines" in the left column
  5. Click "New" (it's in the second column from the left side of the screen) and then "New build pipeline"
  6. Click "GitHub" for the next window which asks "Where is your code"
  7. Select the GEOS-Chem repository
  8. The screen will say "Configure your pipeline". Select "Existing Azure Pipeline YAML file"
  9. It will ask you to select an existing YAML file. Put "feature/ci-pipelines" for the branch and ".ci-pipelines/quick-build.yml" for the Path (this is the Azure pipeline config file).
  10. Repeat 4-9 for ".ci-pipelines/build-matrix.yml" and ".ci-pipelines/release.yml"

Service connections

The "release" pipeline requires a service connection because it automatically pushes the built docker container to Docker Hub. To do this you should

  1. Navigate to the GEOS-Chem project on Azure DevOps (where Step 3 in the instructions above got you)
  2. Click Project settings (in the bottom left) > Service connections (under Project Settings/Pipelines) > New service connection > Docker Registry
  3. Select "Docker Hub" for the "Registry type"
  4. Make the connection name "DockerHub"
  5. Set the Docker Registry to "https://index.docker.io/v1/" if it isn't already
  6. Enter the GEOS-Chem organization's Docker Hub username as the Docker ID
  7. Enter the GEOS-Chem organization's Docker Hub password
  8. Click "Verify this connection"
  9. Click OK

To whoever sets this up: If you would like some help with this when you are setting this up, feel free to let me know and I can make sure I'm on Slack. We could message and/or video call with screen share if that would be helpful.

Release Docker container

The containers that the release pipeline builds have geos-chem-standard, geos-chem-tropchem, geos-chem-soa_svpoa executables in the PATH and their environments are configured. To run GEOS-Chem, the user should

  1. Mount their run directory and ExtData as volumes when they do docker run ...
  2. cd to their run directory (in the docker container)
  3. Call geos-chem-xxxxx where xxxxx is the chemistry they want

Btw, these docker containers are going to be useful for us here at WashU because our new cluster runs docker images (when you log in you specify the docker container you want to load).

Penelope

All of this relies on my penelope Docker Hub registry which is a set of consistent containers for various build environments. By consistent I mean that if you run penelope:0.1.0-ubuntu16.04-mvapich2.3.1-esmf8.0.0 it looks the same as penelope:0.1.0-ubuntu16.04-openmpi3.1.4-esmf8.0.0. This is important so that our CI pipelines can run the same commands in the various containers which makes the build matrix easy.

These containers are built by an Azure pipeline that I set up for LiamBindle/penelope. I'm not convinced going with an Azure pipeline for building/deploying these was the best approach, but it was easy so that's why I did it. It's a bit finicky (it build containers for 35 different build environments every time). I'm not sure if this project should be moved to GEOS-Chem control or spun off as it's own standalone project. Any thoughts?

I think this can be done immediately

Keep in mind that none of this affects the GEOS-Chem repo. I've already added the pipeline YAML files to feature/ci-pipelines and that should be all we need to set up the DevOps project (i.e. this PR doesn't need to be merged before we start using the Azure pipelines). The only public-facing thing is the release containers that will automatically be pushed to Docker Hub upon tagged releases on GitHub.

Looking forward to hearing your thoughts on moving forward.

Liam

@LiamBindle LiamBindle changed the base branch from master to dev/12.7.0 October 24, 2019 20:00
@LiamBindle LiamBindle changed the base branch from dev/12.7.0 to master October 24, 2019 20:08
@LiamBindle LiamBindle changed the base branch from master to dev/12.6.1 October 24, 2019 20:09
@yantosca
Copy link
Contributor

This is great! I can try to set it up on the Github end soon.

@LiamBindle
Copy link
Contributor Author

Great! Let me know if you run into any problems.

@yantosca
Copy link
Contributor

One other question: should we also trigger a quick build for all "feature/*" requests? Sometimes we open feature branches before merging into dev?

@LiamBindle
Copy link
Contributor Author

LiamBindle commented Oct 25, 2019

Sure, that's fine by me.

I went back and forth on this. The reason ended up not adding the trigger on feature branches is because feature branches oftentime have transient changes that break the build (at least major features), and open pull requests and commits to open pull requests already do trigger quick build.

Quick build only takes about 10 minutes though (~5 min overhead and ~5 min to compile), so there isn't really a downside to triggering it frequently

@yantosca
Copy link
Contributor

Ah I see. Maybe it's OK if we only check the dev then. That's when it's really important.

@LiamBindle
Copy link
Contributor Author

If someone does want quick build to check their feature branch, they can open a draft pull request which can't be merged but will trigger quick build.

@yantosca
Copy link
Contributor

I've set up an Azure project under the name GEOSChem and have established a connection to dockerhub. So what is the next step?

Also I think we should ask the GCST to create identities on Azure so I can add them as admins.

@LiamBindle
Copy link
Contributor Author

I'll message you on Slack

@yantosca yantosca merged commit 2b1584d into dev/12.6.1 Oct 25, 2019
@yantosca yantosca deleted the feature/ci-pipelines branch October 25, 2019 15:59
@yantosca yantosca self-assigned this Oct 25, 2019
@yantosca yantosca added the build Related to the build sequence (cmake or make) label Oct 25, 2019
@yantosca yantosca added this to the 12.6.1 milestone Oct 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to the build sequence (cmake or make)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants