Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a cylc suite to run all recipes #2219

Merged
merged 9 commits into from Jul 30, 2021
Merged

Add a cylc suite to run all recipes #2219

merged 9 commits into from Jul 30, 2021

Conversation

bouweandela
Copy link
Member

@bouweandela bouweandela commented Jul 2, 2021

Description

This pull request adds a cylc suite for running all recipes. It also includes a script to generate an overview page for many recipe runs. In the future, this will facilitate recipe regression tests. For now, it can be handy to run all recipes before a release.

Results from the recipe runs are available here


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the πŸ›  Technical or πŸ§ͺ Scientific review.


To help with the number of pull requests:

@bouweandela bouweandela changed the title Run all recipes Add a cylc suite to run all recipes Jul 2, 2021
@valeriupredoi
Copy link
Contributor

I'd be very curios on which cluster this could run, given it'll be a memory/runtime monster πŸ‘Ή Did you want to retire/adapt the old rose-cylc wrapper ?

@bouweandela
Copy link
Member Author

bouweandela commented Jul 9, 2021

I'd be very curios on which cluster this could run, given it'll be a memory/runtime monster πŸ‘Ή

Every recipe runs on it's own node, so as long as the node is fat enough to run the recipe things should work out fine. Most recipes finish quite fast, but there are a few that need more resources, see the recipes listed under Computational issue? in #2198 (comment).

Did you want to retire/adapt the old rose-cylc wrapper ?

It seems very Met Office/Jasmin specific from the documentation, without having access to that it is not at all clear how to use it. Also, what is the advantage of having a wrapper script and using rose instead of just plain cylc? The cylc suite in this pull request is only 20 lines or so.

@valeriupredoi
Copy link
Contributor

yeah that's what I meant, we'll have to specify memory requirements per job (per recipe) and also run time requirements (what was the max duration of a SLURM short serial again?) since if you do a blanket memory/run time for all the joobs will wait in the queue for a loong time. About the old Rose wrapper - shall we retire it then?

Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work, Bouwe!

@bouweandela
Copy link
Member Author

Thanks for reviewing @valeriupredoi! I'll just merge this now and we can add improvements once we get more funding for automating regression tests.

@bouweandela bouweandela merged commit b0a44bd into main Jul 30, 2021
@bouweandela bouweandela deleted the run-all-recipes branch July 30, 2021 13:57
@valeriupredoi
Copy link
Contributor

sounds like a good plan (yeah, good luck with funding 😁 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants