Do you have a recipe that requires a lot of memory or time to run?

[bouweandela]

If you have a recipe that requires a lot of memory or takes a long time to run, please share it here. We will use those to test the new and improved dask support. Further performance optimizations are planned in #2041 and #2042. This works with lazy (preprocessor) functions, see #674 for progress on making preprocessor functions lazy.

You can just attach your recipe to the answer (rename it to .txt so GitHub allows attaching it) or copy/paste it below.

We're specifically interested in improving the (memory) performance of the preprocessing, so any diagnostic scripts can be left out of the recipe if you're not ready to share those.

@ESMValGroup/esmvaltool-developmentteam

[pepcos]

Here is one example, if more come to mind I'll add them.

This one is not super heavy memory-wise but it would be great to have as many processes running at once (which might benefit from computing in multiple nodes as not all machines have the amazing amount of cores per node that Levante has).

recipe_decadal_parallel_tasks.txt

[bouweandela]

Thanks, I'll get back to you once we have some results.

[Peter9192]

The recipe for wflow forcing is also very memory-heavy. That's mostly the diagnostic script, which does use the esmvaltool regridding function. /hydrology/recipe_wflow.yaml. If it helps, I could prep a version that moves the regridding to the preprocessor?

[bouweandela]

Yes, that would be great!

[Peter9192]

There you go. I tried using the lazy regridder from iris-esmf-regrid, not sure if you prefer this or the standard regridder for your optimization exercise. You can easily play with the resolution or the number of years to make it more or less challenging. The current settings reflect our real-world use case.

recipe_wflow_testcase.yml

# ESMValTool
# recipe_wflow_testcase.yml
---
documentation:
  title: Test recipe for performance optimization.
  description: |
    Pre-processes climate data for the WFlow hydrological model.

  authors:
    - kalverla_peter

preprocessors:
  regrid:
    regrid:
      # Ideally we want ~1 km grid spacing with a custom grid definition. This
      # requires a lazy regridder or a huge amount of memory.
      target_grid:
        start_longitude: 5.4
        end_longitude: 11.8
        start_latitude: 46.3
        end_latitude: 51.9
        step_latitude: 0.01
        step_longitude: 0.01
      scheme:
        # Trying to use (experimental) lazy regridder
        reference: esmf_regrid.schemes:ESMFAreaWeighted

diagnostics:
  wflow_daily:
    description: WFlow input preprocessor for daily data
    additional_datasets:
      - {dataset: ERA-Interim, project: OBS6, tier: 3, type: reanaly, version: 1}  # or any other
    variables:
      orog:
        mip: fx
        preprocessor: regrid
      tas: &daily_var
        mip: day
        preprocessor: regrid
        start_year: 2001
        end_year: 2016
      pr: *daily_var
      psl: *daily_var
      rsds: *daily_var
      rsdt:
        <<: *daily_var
        mip: CFday
    scripts: null

[Peter9192]

Also: feel free to use another dataset if that's easier

[bouweandela]

There you go. I tried using the lazy regridder from iris-esmf-regrid, not sure if you prefer this or the standard regridder for your optimization exercise.

Thanks! It doesn't matter for now, just use whatever you would use in the real case.

You can easily play with the resolution or the number of years to make it more or less challenging. The current settings reflect our real-world use case.

That's great, we're most interested in real use cases.

[Peter9192]

Thanks! It doesn't matter for now, just use whatever you would use in the real case.

In the real case we were using the standard iris (area-weighted) regridder.

[bouweandela]

The test runs for the v2.7 release also provide some recipes with large memory usage and poor parallization:

recipe	runtime	memory (GB)	CPU (%)
recipe_collins13ipcc.yml	6:03:30	320.8	131.2
recipe_smpi_4cds.yml	2:10:52	192.7	107.9
recipe_eyring13jgr_12.yml	1:52:15	176.4	107.7

compare the above to this example that seems to parallelize well:

recipe	runtime	memory (GB)	CPU (%)
recipe_daily_era5.yml	0:15:50	106.9	8603.8

and there are more.

[ruthlorenz]

the collins13ipcc was developed before many of the preprocessors were available (for v1), so it would need a rework to make better use of the preprocessors (but one needs to be careful because sometimes they could not be used because of the order of calculations I think to remember).

[bouweandela]

Thanks for the pointer, I already suspected something like that from looking at the recipe. The order of the preprocessor steps can be customized by using custom_order: true in the preprocessor section to run them in the order that they are listed there, so that might help with getting the order of the calculations right. In any case, it might still be interesting to look at this recipe to see if we can at least improve the bits that use the preprocessor.

[malininae]

Do you want an extremely memory consuming recipe which is not in the main yet?

[bouweandela]

Yes, that would be great!

[malininae]

https://github.com/ESMValGroup/ESMValTool-AR6-OriginalCode-FinalFigures/blob/main/IPCC_README_files/README_AR6_WG1_Chap3_Fig3_25_ocean_basins.md was extremely problematic! It was created a while ago, so I think it was still ESMValTool v2.1 or v2.2. I remember it being so bad, so I couldn't process it normally on JASMIN and had to split into two recipes and glue the results together manually after. So good luck!

[bouweandela]

Thanks!