Re-factored GFDL gitlab pipeline #144
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The MOM6 pipeline commands were scripted using make in the file MOM6-examples/tools/MRS/Makefile. This was trying to encapsulate the commands used in the pipeline so they could also be used at the command line (for emulating the pipeline). However, it required emulating the temporary/transient work-spaces provided by gitlab. Also, the Makefile was impenetrable. New approach: - Create a persistent working space for each pipeline, just as you'd work interactively in a single working directory - Use the same commands in .gitlab-ci.yml as we'd use interactively (these still use the tools/MRS Makefile.build and Makefile.run) - The compute stage of tasks is now a bash script (.gitlab/mom6-ci-run-script.sh) which will work/can be submitted from your working directory - The caching of results from each run (gnu/intel/pgi, symmetric/nonsymmetric/layout/etc) is stored locally in the same working directory so we don't have to look elsewhere for the results - Using the "fetch" strategy allows later stages to startup in mere seconds (previously re-cloning the repo took minutes) This does not shorten the turn around significantly but I believe it is a lot easy to follow and emulate. Todo: - split up the run stage to reduce it from 41+ minutes to ~13 minutes.
Codecov Report
@@ Coverage Diff @@
## dev/gfdl #144 +/- ##
=========================================
Coverage 33.45% 33.45%
=========================================
Files 262 262
Lines 71434 71434
Branches 13339 13339
=========================================
Hits 23896 23896
Misses 43064 43064
Partials 4474 4474 Continue to review full report at Codecov.
|
marshallward
approved these changes
Jun 13, 2022
|
Gaea regression: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/15863 ✔️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The MOM6 pipeline commands were scripted using make in the file MOM6-examples/tools/MRS/Makefile. This was trying to encapsulate the commands used in the pipeline so they could also be used at the command line (for emulating the pipeline). However, it required emulating the temporary/transient work-spaces provided by gitlab. Also, the Makefile was impenetrable.
New approach:
This does not shorten the turn around significantly but I believe it is a lot easy to follow and emulate.
An example pipeline is https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/15815 (GFDL-only)
Todo: