-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DISCUSSION] GCHP needs a Continuous Integration (with a build matrix) #43
Comments
Better wait until the CMake update? @LiamBindle probably knows better. |
The build matrix would be especially useful. That would help us identify versions of our dependencies and combinations of dependencies (if any) that are broken. That would help simplify the "GCHP Quick Start Guide" which is a bit daugnting right now. It would probably be best to wait until the CMake update IMO for the following reasons: (1) gchp_ctm is dropping support for GNUMakefiles so we will have to set up the CI again in a few months, (2) ESMF becomes an external library which significantly reduces compile-time and the build's memory requirements, and (3) set up with CMake should be faster because there is less reliance on environment variables. I'll look into getting this going for gchp_ctm. |
Maybe as a first step we could try to set up a build matrix for GC-Classic, just to get going. That would use everything except the MPI. Then we could translate that over to GCHP once the Cmake transition there is complete. Just a thought... |
That's a good idea. If no one else plans to take this on (or needs it urgently) I could start looking at it in my downtime. It sounds fun side project. I'm pretty busy right now though, so I'm not sure when I'll be able to get around to it. Are you looking at picking this up @JiaweiZhuang? |
@LiamBindle, I might be able to help you out with this as well, as time allows. If we make a container, we should store it on the GCST dockerhub site so that we can all have access to it. I've been recently trying to fix a lot of the GCPy issues and make updates that were requested by the GCSC to the benchmark output. But if you need a hand I could probably find some time. |
@yantosca, sounds good. I'll keep you posted on any progress I make then. |
It will be a good first exercise! I guess testing different GCC versions would be useful. But the biggest use case for this is indeed making GCHP build more robust. Building GC-classic is easy. |
That would be wonderful @LiamBindle ! I would recommend trying Azure pipeline first as it seems quite popular right now. I am also going to use it for my package. |
Hi everyone, Disclaimer: This is just me spilling my thoughts/findings from last night I started looking into this last night and I think it looks pretty straightforward. I thought I'd put my finding here so that others with more expereince and knowledge (@JiaweiZhuang, @yantosca, and others) with docker and CI could potentially recongize any rabbit holes/antipatterns in my plan. Azure pipelinesI found it really easy to set up a simple azure pipeline with a matrix. Thanks for the suggestion @JiaweiZhuang! The documentation is great, and also, their youtube tutorials made setting up an account and project really easy. (e.g. this one, this one, and this one ). I set up a basic trigger:
- master
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
gcc4:
containerImage: gcc:4
gcc5:
containerImage: gcc:5
gcc6:
containerImage: gcc:6
gcc7:
containerImage: gcc:7
gcc8:
containerImage: gcc:8
container: $[ variables['containerImage'] ]
steps:
- script: |
gfortran --version
mkdir build
cd build
cmake -DRUNDIR=IGNORE -DRUNDIR_SIM=standard $(Build.Repository.LocalPath)
make -j
make install
displayName: 'Building GEOS-Chem' Here is the pipeline project: https://dev.azure.com/lrbindle/geos-chem/_build?definitionId=1. Essentially, my plan is to just replace the GEOS-Chem-Build-Matrix ImagesI put together a simple dockerfile that builds an image with GEOS-Chem Classic's dependencies. You can find it here. Essentially, I was thinking that I'd set up an azure pipeline with a matrix of Further stepsFirst, I'm going to get the "geos-chem-build-matrix" pipeline going. I thought I would start with building images with all gcc major versions, and the latest HDF5, NetCDF C and Fortran libraries. I'm planning to push these to DockerHub/liambindle/gcc-netcdf-c-netcdf-fortran with tags like "5-4.7.1-4.4.5" for thegcc, netcdf-c, and netcdf-fortran versions. The second step will be creating a branch on GEOS-Chem like Next week I'm going to be focusing on prepping for our visit to Goddard, so after today, it will probably be about 2-3 weeks before I can pick this up again. I thought I should get this down to make picking it back up in a few week easier. |
This is great, Liam! I think using Azure pipelines is a good move. |
@LiamBindle That's really fast work!
Oh, I didn't notice this before. They have some great stuff.
Great, just quick comments: LiamBindle/geos-chem-build-matrix#1 LiamBindle/geos-chem-build-matrix#2
I think we should reuse Docker images more cleverly. The build matrix can become every large. We might end up having something like:
That's 2x3x8=48 parallel builds. Alternatively, we can pick up some representative combinations, instead of try all possible cases. But 20+ builds would still be normal. Some points to consider:
|
Thanks for the feedback! @JiaweiZhuang, thanks, I thought you would probably have some good insight because I think you have a lot more experience with containers than I do. I think it maybe wasn't quite clear though in what I'm thinking.
This is what I am thinking. Essentally, LiamBindle/geos-chem-build-matrix is a project that builds and pushes such images to DockerHub. We can pull these images later in geoschem/geos-chem for our build matrix pipeline. Because there are many possible combinations, as you have mentioned, I'm working on a pipeline that uses a matrix to build all these prebuilt images and push them to dockerhub. Once those are working, we can set up a Azure Pipeline for geoschem/geos-chem that pulls these prebuilt images with a matrix. This should be easy once there are images we can just pull from Docker Hub. I think that when I get the prebuilt image pipeline working it would be good to make the images more complex (e.g. seperate images for each gcc, and then install multiple netcdf versions in each image by using spack). What do you think? |
Oh I thought you will be building the images on the fly before building GEOS-Chem (didn't see DockerHub-related commands in your repo). But now I see your plan. From Azure docs "Docker task" section I see that:
The Docker Registry service connection can be either Azure Container Registry or Docker Hub. If I understand correctly, this "Docker task" is mainly for publishing/deploying/pushing Docker images (say, to get around the resource limit of DockerHub's own build). Then, the next stage, is using Container jobs to pull pre-built images (containing MPI libraries, etc.) to further build GCHP. |
Basically, there are two independent steps 1. Build the base images containing NetCDF and MPI libraries. This can be done on DockerHub itself, or on Azure pipelines, or on many other platforms (even on local machines). This is mostly what your current repo (https://github.com/LiamBindle/geos-chem-build-matrix) is doing. This step only needs to be done once, and rarely needs a rebuild. One major question for this step is how to manage all variants of images and minimize redundant installation of libraries. A useful reference for handling image dependencies is Jupyter's docker stack: https://github.com/jupyter/docker-stacks (it has a deep dependency chain) This step needs its own repo. Maybe eventually merging it to https://github.com/geoschem/geos-chem-docker. 2. Actually run CI inside the images built in step 1 This is where the "build matrix" is defined. Although you can still have a "build matrix" in step 1, it won't be as big as the matrix here (a single image might contain multiple libraries) Azure pipelines will be the primary choice, as it provides more resources than Travis & Docker Hub. This step needs to add CL config files into GEOS-Chem's code repo, so the build can be triggered at every commit. GCHP's code repo should keep a tag/hash pointing to a GC-classic commit. |
In terms of the resource limit on Azure pipeline, I saw in Parallel jobs that:
So the matrix can't be too big if we stick to the free plan :) But 6 hours are very long, compared to 50 minutes on Travis and 2 hours on Docker Hub. A single job can probably finish 3~4 GCHP builds (another reason why merging multiple libraries into one image is useful) As a starting point, let's just build 8~10 environments independently, choosing from:
|
@LiamBindle brought some good ideas in a conservation.
For GCHP, can reduce the number of compiler variants (older |
Hi everyone, Just following up on where I got earlier today. Pipeline for generating build matrix imagesI got a initial version of a pipeline for building the build matrix images done and that can be found here: LiamBindle/geos-chem-build-matrix. Because it was easiest, this inial version only builds the following images: latest netcdf-c and netcdf-fortran with gcc4, gcc5, gcc6, gcc7, gcc8, and gcc9, as well as netcdf4.1 with gcc7. These should be a good start, and we can build on these to cover multiple versions of glibc and different base OSs as next steps. A build matrix pipeline for GEOS-Chem ClassicAs a test, I set up a build matrix pipeline on the I wasn't able to do this for on a feature branch of the offical GEOS-Chem repo because I don't have the proper permissions. To set this up for GEOS-Chem, we just need to create a GEOS-Chem organization on Azure DevOps, create a GEOS-Chem project, create a new pipeline, and then replace the default I'm going to be focusing on prepping for our Goddard visit for the next week, but I'll be happy the help set this up once I'm back. |
@LiamBindle Thanks for the update!! Quick question: I cannot access the link https://dev.azure.com/lrbindle/geos-chem/_build/results?buildId=35. Got "401 - Uh oh, you do not have access." after logging into my Azure account. Would you be able to make the link public (without requiring log-in)? |
Whoops, sorry about that. Done! |
Just notice LLNL's Building-Linking-and-Testing (BTL) framework (https://github.com/LLNL/blt). Not sure if that's useful for our task, but @LiamBindle might find it interesting:
|
@JiaweiZhuang Thanks for mentioning. I've come across BLT before, but personally, I don't think a CMake macro libraries like BLT would benefit GEOS-Chem/GCHP. The new MAPL already depends on ecbuild. It might be that larger projects find these libraries convenient, but I think that for GEOS-Chem, vanilla CMake is more clear and easier to maintain—for now at least. |
@LiamBindle Just notice a paper on CI/CD for HPC using Jenkins and Singularity container: Continuous Integration and Delivery for HPC: Using Singularity and Jenkins Following a similar idea, it is totally possible to develop a build-run-plot CI pipeline to automate the entire GEOS-Chem benchmarking workflow, with additional benefits like performance monitoring (see pydata benchmark for example). For the "delivery" side, the same pipeline can be further extended to build containers/AMIs/Conda packages/Spack packages for GC-classic/GCHP, for general science users. CI/CD is rarely done for huge HPC codebase, but I think you have the technical capability to make it work, and then the same workflow can be adopted by other models. It can become a serious research project and lead to a GMD-style publication, if you'd like to continue working on build systems. References: The actual framework (AWS vs Azure vs open-source) probably doesn't matter that much, because the high-level logic of those tools are very similar, and they will face the same challenges such as how to efficiently pull GEOS-Chem's large input data into a container environment. |
That's very cool. Thanks for the info! I think run/plot pipelines would be pretty doable with a self-hosted agent or two. Self-hosting agents might also make ifort licensing easier. Right now though, I don't think I have the capacity to take something like this on, but I think this would be a really cool project for someone to pick up! I'm going to submit a CI PR for GEOS-Chem Classic this afternoon. Looking forward to your hearing everyones feedback. |
Indeed. That CI/CD paper also runs on-prem, and that's why they use Singularity instead of Docker due to permission issues on local HPC. Running the pipeline on Harvard Odyssey/Cannon will simplify data movement, but that also has many downsides such as limited compute resources, frequent system maintenance and update, etc.
Yeah I just put the ideas here. Nothing urgent at all! |
In today's telecon, @sdeastham suggested using fake/null variables in ExtData to minimize the input data size, so we can actually run the model (not just build) as part of the CI pipeline. This can catch many of GCHP's run-time errors that cannot be detected at build time. Long time ago I tried putting As long as we can shrink the input data to less than 1 GB, we can package those data into a container image, without having to write scripts to pull input data on the fly. Different from a real benchmark, the simulation here doesn't have to be scientific meaningful, as the goal is just to catch bugs. Running a real benchmark on CI will be a good next step, but let's just get a synthetic simulation running for now. |
From the MAPL manual dated 2014 draft:
|
Thanks @lizziel ! Is there a similar feature in HEMCO for GC-classic? |
Continuous integration (build only) is available with GCHP 13.0.0. I will therefore close this issue. However, some of the discussion here is relevant for continuing to develop the CI capability to include running the model. I am therefore porting this issue to the GCHPctm repository which replaced this GCHP repository with the release of 13.0.0. |
Here are some sample configurations for using intel compilers in public CI systems : https://github.com/oneapi-src/oneapi-ci |
Problem
The difficulty to build GCHP (despite the large improvement over early versions) is preventing user adoption and eating a lot of engineer time (e.g. on debugging makefiles). In particular, it is time-consuming to diagnose compiler/MPI-specific problems, as there are so many combinations of them.
So far this problem is being treated passively -- we stick to very few combinations we know that is working (notably ifort + OpenMPI3). Other combinations (gfortran, other MPIs) are solved case-by-case, typically when a user hit bugs on a specific system.
Suggestions
The sustainable way (which will save a lot of engineer time in the long-term) is to deal with this problem actively -- we should explore all common combinations using build matrix that is offered by most Continuous Integration (CI) services.
The components of the build matrix include:
(for issues like [FEATURE REQUEST] Restrict mass balance operation to the troposphere #17)
(for issues like Feature/hemco submodule #15 and many other compiler problems)
(for issues like Fixes GCHPctm's build's "all" target and installation #9 [BUG] -DCMAKE_BUILD_TYPE=Debug Fortran_FLAGS missing comma #35 and many other MPI problems)
By having a continuous build at every commit / every minor releases, we will be able to:
This also helps user to find the "shortest path" to solve their specific error. An example question is "my build is failing on Ubuntu + gfortran + mpich; which component should I change to fix the problem"? By looking at the matrix, you can see that (for example) changing the MPI can lead to a correct built.
Where to start
A simple CI (on Travis) for GC-classic is geoschem/geos-chem#11 However, the memory & compute limit on Travis probably won't allow building GCHP. Other potentially better options are:
Tutorial-like pages:
Existing models for reference:
The text was updated successfully, but these errors were encountered: