shell
Note
This is a feature aimed for advance users and developers who work in preparing default configurations of experiments, or who implement a model/coupled-setup in ESM-Tools.
ESM-Tests is the integration testing suite from ESM-Tools. Its aim is to test a set of selected runscripts and model builds just by using one single command: esm_tests
. It can also perform dry actions (i.e. check compilations and check runs).
- actual vs check test
An actual test is a test where the model has been compiled and run in one of the supported HPCs. A check test is a dry test, meaning, no compilation or run takes place, but instead, the configuration files and scripts are generated. Both actual and check tests compare their output configuration files and scripts to the last-state, and offer the possibility to update the last-state of those files at the end of the test.
- esm_tests_info
The repository where the files of the
last-state
and therunscripts
for testing are stored. This repository is clone as a submodel of ESM-Tools wheneveresm_tests
command is run for the first time. The repository is cloned locally into theesm_tools/src/esm_tests/resources
folder. You can activate the submodule manually viagit submodule init
followed bygit submodule sync
.- last-state
Set of configuration files, both from compilation and runtime, that represent the last approved configuration of the testing runscripts. This set of files is kept for comparison with the equivalent files of future pull-requests, to ensure the stability of the configurations. ESM-Tests always compares the new files to the
last-state
files automatically, both in actual compilation/runs or check compilation/runs. Seeesm_tests:Last-state
.- runscripts
The runscripts to run the tests. Runscripts define the test simulation details as in regular ESM-Tools runscripts, but are also used for ESM-Tests to understand what needs to be compiled. Runscripts are part of the
esm_tests_info
submodule, and can be found (if the submodule was initiated viaesm_tests -u
) inesm_tools/src/esm_tests/resources/runscripts
. Runscripts need to be generalized (i.e.choose_computer.name
) for the different HPCs where you want to run the tests.state.yaml
In this document some times referred only as state, is a YAML file that includes information about the status of the tests (actual tests or check tests) in different computers. It also includes the date of the last actual test.
ESM-Tests is designed to compile and run tests just with one single command, without additional arguments: esm_tests
, so that launching a suite of tests in a supported HPC is straight forward. Higher granularity in the control of the tests is enabled via:
esm_tests:Arguments
- Runscripts via the usual
esm_parser
syntax (e.g.choose_computer.name
) esm_tests:Model control file (\`\`config.yaml\`\`)
esm_tests:Local test configuration (\`\`test_config.yaml\`\`)
The commands syntax is as follows:
esm_tests [-h] [-n] [-c] [-u] [-d] [-s SAVE] [-t] [-o] [-b] [-g] [-e] [-r BRANCH]
Optional arguments | Description |
---|---|
|
Show this help message and exit |
|
Avoid loading user config (for check tests in GitHub) |
|
Check mode on (does not compile or run, but produces some files that can be compared to previous existing files in |
|
Updates the resources with the release branch, including runscriptsand last_tested files |
|
Delete previous tests |
|
Save files for comparisson in |
|
Print the state stored in state.yaml |
|
Hold before operation, to give time to check the output |
|
Bullet points for printing the state and copy/paste as markdown text |
|
Use this flag when running in GitHub servers (i.e. adds syntax for collapsing compare sections of the output for GitHub Actions) |
|
Trigger a system exit on errors or file differences so that GitHub actions can catch that as a failing test |
|
use the given esm_tests_info branch |
The last-state
files are https://github.com/esm-tools/esm_tets_info repository, in the release
branch. The files stored in the last-state
are: * compilation scripts (comp-*.sh
) * namelists * namcouple * finished_config * batch scripts (.run
)
As a user, you can check the last-state
status (the online one of the esm_tests_info
repo, release
branch) by running:
esm_tools --test-state
This will give you a summary of the state of compilation and running tests for different models, in different computers, and also a date of when the latest actual compilation and run tests were carried out.
If you are testing locally in an HPC, you can get the same information about your local state by running:
esm_tests -t
File location: esm_tools/src/esm_tests/resources/runscripts/<model>/config.yaml
Versioned: Yes, distributed with esm_tests_info
The Model control file gives you control over ESM-Tests setups for the set of runscripts for a given model (the model which name is the same as the folder where the config.yaml
is contained: esm_tools/src/esm_tests/resources/runscripts/<model>/
).
Within this file you can control:
- which files need to be present for considering an
actual compilation test
successful (comp.actual.files
) - which files need to be present for considering an
actual run test
successful (run.actual.files
) - which messages from the execution of
esm_runscripts
should trigger an error in anactual run test
(run.actual.errors
) - which
computers
are supported for this set of tests (computers
)
The file should contain this structure:
comp:
actual:
files:
- "file/path" # Typically the binaries
check: {}
run:
actual:
errors:
- "error message to mark the test as not successful # Typically "MISSING FILES"
files: # Typically restart files and outdata files
- "path/to/file1"
- "globbing/path/*/to*files"
check: {}
computers:
- <computer1>
- <cimputer2>
In the files
sections, globbing is supported.
The file's paths should be relative to the compilation folder or the experiment folder.
Each file name can be followed by the syntax in/except [<model_version1>, <model_version2>, ...]
to only check for that file in
that set of model versions, or to exclude (except
) that file from being check for a set of model versions.
Example
comp:
actual:
files:
- "bin/fesom*"
- "bin/oifs"
- "bin/rnfma"
check: {}
run:
actual:
errors:
- "MISSING FILES"
files:
- "restart/fesom/fesom.*.oce.restart/hnode.nc*"
- "restart/fesom/fesom.*.oce.restart/salt.nc*"
- "restart/fesom/fesom.*.oce.restart/ssh_rhs_old.nc*"
- "restart/fesom/fesom.*.oce.restart/temp.nc*"
- "restart/fesom/fesom.*.oce.restart/urhs_AB.nc*"
- "restart/fesom/fesom.*.oce.restart/vrhs_AB.nc*"
- "restart/fesom/fesom.*.oce.restart/w_impl.nc*"
- "restart/fesom/fesom.*.ice.restart/area.nc*"
- "restart/fesom/fesom.*.ice.restart/hice.nc*"
- "restart/fesom/fesom.*.ice.restart/hsnow.nc*"
- "restart/fesom/fesom.*.ice.restart/ice_albedo.nc*"
- "restart/fesom/fesom.*.ice.restart/ice_temp.nc*"
- "restart/fesom/fesom.*.ice.restart/uice.nc*"
- "restart/fesom/fesom.*.ice.restart/vice.nc*"
- "restart/oasis3mct/rmp_*"
- "restart/oasis3mct/rstas.nc*"
- "restart/oasis3mct/rstos.nc*"
- "restart/oifs/*/BLS*"
- "restart/oifs/*/LAW*"
- "restart/oifs/*/rcf"
- "restart/oifs/*/srf*"
- "restart/oifs/*/waminfo*"
- "outdata/oifs/*/ICMGG* except [frontiers-xios, v3.1]"
- "outdata/oifs/*/ICMSH* except [frontiers-xios, v3.1]"
- "outdata/oifs/*/ICMUA* except [frontiers-xios, v3.1]"
- "outdata/oifs/atm_remapped* in [frontiers-xios, v3.1]"
- "outdata/fesom/*.fesom.*.nc"
check: {}
computers:
- ollie
- mistral
- juwels
- aleph
- blogin
- levante
File location: esm_tools/src/esm_tests/test_config.yaml
Versioned: No, user specific, git-ignored
This file gives you control on which tests esm_tests
will run in the current machine, independently of what tests are defined in the Model control files. The current machine needs to be included in the Model control file for the test to run (i.e. test_config.yaml
runs only the tests included there and supported on the current platform). The syntax is as follows:
<model1>: - <runscript1_name>.yaml - <runscript2_name>.yaml - [ ... ] <model2>: all [ ... ]
The model
sections need to be named after the folders in esm_tools/src/esm_tests/resources/runscripts
. If you want to run all the suported runscripts for a model in this platform, make the model
section have the value all
. If you want to select a set of supported runscripts make the model
be a list of runscripts (this runscripts need to be in esm_tools/src/esm_tests/resources/runscripts/<model>/
). If you want to run all the supported runscripts for all supported models in this platform, but still keep this file around (i.e. commented most of the contents), make the file content be an empty dictionary ({}
).
Example
#{} awiesm: #all - all_awiesm-2.1-recom.yaml # - awiesm-2.1_icebergs.yaml - bootstrap.yaml - pico.yaml - PI_ctrl_awiesm-2.1-wiso.yaml - pi.yaml - pi-wiso.yaml echam: all fesom: all awicm: all # - awicm1-CMIP6-initial-monthly.yaml # - awicm2-initial-monthly.yaml fesom-recom: - fesom-recom1.4-initial-daily.yaml awicm3: all # - awicm3-v3.1-TCO95L91-CORE2_initial # - awicm3-frontiers-TCO159L91-CORE2_initial.yaml #oifsamip: all #vilma-pism: all
- Add the given runscript to
esm_tools/src/esm_tests/resources/runscripts/<model>/
- Make sure your runscript has a meaningful name
- Make sure your runscript has the correct model
version
defined, for a standalone model in the section of the model (not ingeneral
), and for a coupled setup, both in thegeneral
section and in the coupled setup section (e.g.awiesm
section). This version will be used by ESM-Test for theesm_master
command to compile Modify the following variables to take the environment variables setup by `ESM-Tests`:
general: account: !ENV ${ACCOUNT} base_dir: !ENV ${ESM_TESTING_DIR} <standalone-model/setup>: model_dir: !ENV ${MODEL_DIR}
- Generalize the runscript to be able to run in the computers where you'd want it to be supported (i.e. add the necessary
choose_computer.name
switches) - Create the Model control file (
esm_tools/src/esm_tests/resources/runscripts/<model>/config.yaml
). See ref:esm_tests:Model control file (``config.yaml``) for details about the content - If you are using the
Local test configuration (\`\`test_config.yaml\`\`)
to exclude some models, make sure the current model is included, so that your tests can be run locally.
- In the corresponding Model control file (
esm_tools/src/esm_tests/resources/runscripts/<model>/config.yaml
), add the name of the platform to thecomputers
list - In the runscripts (
esm_tools/src/esm_tests/resources/runscripts/<model>/<runscript>.yaml
), add the necessary case to thechoose_computer.name
to specify pool directories, forcing files,nproc
, etc.
- In the pull-request, if all the tests passed you don't need to approve any changes, you can jump directly to step 4.
If any of the tests labelled as
esm_tests
failed (click on the triangles to expand screen captures):Scroll up and expand the lines starting with COMPILE/SUBMITTING (for compilation and runtime checks respectively) followed by the script with differences
If there are no problematic differences, and the pull-request has been already reviewed and is just ready to be merged, write a message on the PR containing
#approve-changes
. This will commit the new files from the tests as thelast-state
, in theesm_tests_info
repository.Warning
Currently,
#approve-changes
does not update the test status on GitHub, once the operation finishes. If you want to see whether#approve_changes
finished or not you have to navigate to theActions
tab in GitHub. If you want to see all tests green, wait until#approve-changes
finishes, and relaunch the tests for the last failed set of tests in the PR. Miguel - I know this is a pain, but I could not figure out how to do all this automatically (I wasted enough time on GitHub Actions for years to come).- Bump the version and wait that the bumpversion commit shows up.
- You can now merge.