Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a realistic example recipe #3356

Merged
merged 18 commits into from Oct 26, 2023
Merged

Add a realistic example recipe #3356

merged 18 commits into from Oct 26, 2023

Conversation

bouweandela
Copy link
Member

@bouweandela bouweandela commented Sep 12, 2023

Description

Add an example recipe that reproduces part of figure 9.3a from IPCC AR6. This is the figure produced by the recipe:

IPCC_AR6_figure_9 3a_1850-2100

To run the recipe, use the ESMValCore development version main branch or the upcoming [v2.10 release] once it is available.

I used these settings for the Dask distributed scheduler:

~/.esmvaltool/dask.yml:

cluster:
  type: distributed.LocalCluster
  n_workers: 24
  threads_per_worker: 2
  memory_limit: 8GiB

batch job

#!/bin/bash -l 

#SBATCH --job-name=recipe_easy_ipcc.%J
#SBATCH --output=/home/b/b381141/recipe_easy_ipcc.%J.out
#SBATCH --error=/home/b/b381141/recipe_easy_ipcc.%J.err
#SBATCH --account=bk1088
#SBATCH --partition=compute
#SBATCH --time=08:00:00 
#SBATCH --mem=256G

set -eo pipefail 
unset PYTHONPATH 

. /work/bd0854/b381141/mambaforge/etc/profile.d/conda.sh
conda activate esmvaltool-v2.9.x-2023-07-04

esmvaltool run examples/recipe_easy_ipcc.yml --max-parallel-tasks=1 --remove-preproc-dir=False --search-esgf=always

and the run took about 4.5 hours. Running with --max-parallel-tasks=1 is required to avoid overloading the Dask scheduler with too many tasks. As it is, the Dask graph for computing a single output file contains almost 800K tasks. See SciTools/iris#5455 for a discussion of some of the computational performance issues.


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the πŸ›  Technical or πŸ§ͺ Scientific review.

New or updated recipe/diagnostic


To help with the number of pull requests:

@bouweandela bouweandela added the new recipe Use this label if you are adding a new recipe label Sep 12, 2023
@bouweandela bouweandela added this to the v2.10.0 milestone Sep 12, 2023
Copy link
Contributor

@Peter9192 Peter9192 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Only thing we might still want to add is some documentation (including the nice output figure). Perhaps under https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html, or do you think it should be added under https://docs.esmvaltool.org/en/latest/recipes/index.html#ipcc? Let me know if I can help with that.

@bouweandela bouweandela marked this pull request as ready for review October 25, 2023 17:05
Copy link
Contributor

@Peter9192 Peter9192 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, this seems complete to me now πŸ‘

Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

top job, fellas! 🍺 Tho sad to see a "realistic" example recipe taking 5 hours 😁

@valeriupredoi valeriupredoi merged commit 604f6fb into main Oct 26, 2023
6 checks passed
@valeriupredoi valeriupredoi deleted the easy-ipcc-recipe branch October 26, 2023 16:14
@bouweandela
Copy link
Member Author

Over the coming year we will get support from the ESiWACE3 project to make it faster, so if things go according to plan it should be a lot faster soon πŸš€

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new recipe Use this label if you are adding a new recipe
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants