Recipe Test Workflow (RTW) prototype

[KatherineTomkins]

Introduction

• This issue is a summary issue for the development of the Recipe Test Workflow (RTW) prototype
• The requirements for the RTW are available at Recipe test workflow proposal #2723
• Please get in touch with one of the current developers / reviewers (@ehogan, @alistairsellar, @mo-gill and @chrisbillowsMO) if you would like to collaborate / contribute (thanks to previous contributors @KatherineTomkins, @mo-tgeddes and @Jon-Lillis!) 👍

How we are working

• We created a main feature branch for the prototype (called recipe_test_workflow_prototype), and sub-feature branches are used for individual development tasks (see the discussion we had with V)
• We created a Recipe Test Workflow (RTW) label, which we add to every issue and PR
• We open an issue for each individual development task, stating Part of #2786 in the first comment of the issue
• We create a draft PR and link to the issue for the individual development task as normal
• We ensure the RTW checks pass before requesting a reviewer
• Documentation exists but is currently separate from the main documentation (it is located in the recipe_test_workflow directory) and:
  • we hope to add the documentation to main documentation before the main feature branch is merged
  • we update the documentation for each individual development task, as appropriate

Completed Issues / PRs

Task	Issue	PR	Assignee	Merge date
Create a feature branch in the ESMValTool repository, add initial commit	N/A	#2787	@mo-tgeddes	2022-09-02
Update configure task to use the correct paths	#2814	#2815	@KatherineTomkins	2022-09-29
Update the install_cold task to install the latest version of ESMValTool and ESMValCore	#2821	#2803	@KatherineTomkins	2022-10-03
Add compare task to RTW	#2819	#2820	@mo-tgeddes	2022-10-04
Add workflow checks to RTW	#2857	#2859	@mo-tgeddes	2022-10-17
Add two groups of two test recipes	#2816	#2822	@KatherineTomkins	2022-10-20
Upgrade the RTW to work with ESMValTool v2.8.0	#3168	#3169	@ehogan	2023-06-26
Implement RTW review comments	#3245	#2863	@mo-tgeddes	2023-06-28
Separate install cold and clone task for RTW	#2870	#2871	@mo-tgeddes	2023-06-28
Update the RTW to work at the MO	#3385	#3386	@ehogan	2023-10-18
Add JASMIN as an additional site to the RTW	#3060	#3302	@ehogan	2023-10-20
Simplify configure task in recipe test workflow	#3398	#3394	@alistairsellar	2023-10-26
Use Docker containers in the RTW	#3246	#3399	@alistairsellar	2023-12-12
Run RTW prototype daily with local KGO	#3493	#3556	@alistairsellar	2024-03-27
Add imagehash package as an ESMValTool dependency	#3436	#3557	@alistairsellar	2024-04-03
Set resources for RTW recipes individually	#3561	#3562	@alistairsellar	2024-04-09
RTW should support recipes in subdirectories	#3566	#3567	@alistairsellar	2024-05-09
Update Cylc version in lock file for RTW	#3593	#3596	@mo-gill	2024-05-20
Store RTW files in share directory in installed workflow	#3434	#3600	@mo-gill	2024-05-21
Add recipe recipe_consecdrydays to RTW	#3608	#3609	@mo-gill	2024-05-22
Validate RTW user config file	#3392	#3615	@chrisbillowsMO	2024-05-24
Remove imagehash install from RTW	#3621	#3622	@mo-gill	2024-05-29
Add Rose and Cylc to testing dependencies for RTW	#3623	#3625	@ehogan	2024-05-30
Add recipe_ocean_multimap to RTW	#3626	#3627	@chrisbillowsMO	2024-06-10
Replace the MO-specific URL in the RTW with a public URL	#3654	#3657	@mo-gill	2024-06-20
Include verbose output from compare.py in RTW	#3669	#3671	@ehogan	2024-06-21
Add code owners for the RTW	#3667	#3668	@ehogan	2024-06-21
Add rose macro -V command to rtw-tests.yml	#3632	#3649	@chrisbillowsMO	2024-06-24
Add how to add a recipe documentation	#3074	#3614	@mo-gill	2024-06-25
Add the recipe_ocean_amoc.yml recipe to the RTW	#3663	#3664	@ehogan	2024-06-25
Review rose-app.conf files in RTW	#3571	#3601	@mo-gill	2024-06-25
Merge the RTW documentation with the ESMValTool documentation	#3253	#3684	@ehogan	2024-06-26
Final sanity check and reorganisation of RTW	#3685	#3707	@ehogan	2024-07-02

Work to be done after the prototype is merged

Task	Issue	PR	Assignee
Add remaining recipes to RTW
Add a weekly tests cycle to RTW	#3570
Find compare.py more robustly within container	#3437		(see ESMValGroup/ESMValCore#2381)
compare.py should fail cleanly when imagehash not available	#3433
Investigate the use of containers in the RTW at the MO	#3579
Get RTW to check for changes before running tests	#2872
Add a mechanism for notification emails / opening new issues via Github API
Decide how to manage and store KGOs
Decide whether to resolve issues with recipes that are not reproducible
A question about fonts for R diagnostics	#3396
Flag failing recipes and don't re-run until they have been changed or the repo has been updated

Addresses the user stories:

As a release manager
I want to know if changes to main in ESMValTool or Core have caused the recipes to fail to run
So that the API can be fixed before a release

As a release manager
I want to know if changes to ESMValTool or Core have changed the output the recipes
So that I can make fixes or changes before a release

As a recipe owner
I want to know if changes to ESMValTool or Core have changed the output for my recipe
So that I can make fixes or changes before a release

As a member of the ESMValTool or Core development community
I want to run the recipe run workflow efficiently
So that costs are not excessive

As a member of the ESMValTool or Core development community
I want to know about tools available for regression testing
So that I am aware of my responsibilities as a developer

As a member of the ESMValTool or Core development community
I want to know if changes to main in ESMValTool or Core have caused the recipes to fail to run
So that I can fix issues in the API