Generate climatology on the fly for AutoAssess soil moisture #3197

alistairsellar · 2023-05-26T18:44:54Z

Description

Currently the AutoAssess soil moisture recipe (recipe_autoassess_landsurface_soilmoisture.yml) only runs on Jasmin because it requires a climatology which is not available on other machines. This PR makes the recipe portable by generating the climatology on the fly from Tier 2 ESA-CCI data.

In this branch, the climatology is instead made using the ESMValTool seasonal pre-processor. Accessing the pre-processed data requires the script to have direct access to the ESMValTool configuration dictionary, and therefore the soil moisture diag script no longer uses the AutoAssess wrapper script (which translates config information to an alternative dictionary). This makes the recipe a more typical ESMValTool recipe, which in any case aligns with the wider strategy for porting AutoAssess.

The climatology produced by the ESMValTool preprocessor has much greater spatial coverage (fewer points masked out) than the original climatology. At points common to both climatologies the values are very similar but the change in coverage does impact on the metrics, because different regions have different errors. More detail in the comments below to verify that the changes in results are due to the coverage change only. This will need science review, and I am discussing with relevant experts at the Met Office (though others welcome also to review if interested).

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 Any changed dependencies have been added or removed correctly
🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

🧪 Recipe runs successfully
🧪 Recipe is well documented
🧪 Figure(s) and data look as expected from literature
🛠 Provenance information has been added

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

Required in order to access preprocessed obs via main esmvaltool config, and in any case our strategy is now to port autoassess metrics as "native" esmvaltool code. Component changes: - name diag_script directly in recipe - add main() function to handle config and extract filenames - update land_sm_top() to use these preprocessed files

alistairsellar · 2023-05-26T18:46:22Z

@valeriupredoi don't drop your pint 🍺, but I finally made good progress on fixing #2309. I wonder if you'd be interested in tech reviewing it?

alistairsellar · 2023-05-26T18:50:05Z

Science impact

This will need science review, and I am discussing with relevant experts at the Met Office (though others welcome also to review if interested).

There are two science questions regarding the change in results:

Are the climatologies consistent at points where they both have data? Or have changes in source data version or missing data tolerance meaningfully altered the values?
Are the changes in results purely down to the difference in climatologies, or has some difference emerged from the other code changes required for this PR?

Attached (soil_moisture_clims.pdf) is a comparison addressing the first question, the conclusion of which is that the original and new on-the-fly climatologies are consistent in the regions for which they both have coverage.

alistairsellar · 2023-05-26T18:59:53Z

To check the second science question, I tested the impact of changing the climatology alone. This was done by running the original recipe using the main branch at ESMValTool 2.8.0 (so no changes to the diag script), and pointing the recipe at the climatology produced by the ESMValTool pre-processor in runs of this PR. The results are as follows:

Metric	Main branch (control)	main, with preproc clim from this PR	this PR
MedAbsErr djf	0.481511652469635	0.9072856903076172	0.9071897864341736
MedAbsErr mam	0.47429555654525757	0.9059286117553711	0.9060649871826172
MedAbsErr jja	0.6323282718658447	0.7314185500144958	0.731833815574646
MedAbsErr son	0.4487602114677429	0.808375358581543	0.8084797859191895

Thus the changes come from the climatology.

alistairsellar · 2023-05-26T19:02:38Z

@valeriupredoi don't drop your pint beer, but I finally made good progress on fixing #2309. I wonder if you'd be interested in tech reviewing it?

@valeriupredoi I now see the Codacy errors - I'll fix those next week.

Last one requires a minor refactor

Fixes problem of too-many-locals (Alternative approach would be to find another pub?)

...since diag_script is called directly

alistairsellar · 2023-06-02T11:58:46Z

OK, that's the Codacy errors fixed @valeriupredoi, and also documentation updated, so I think ready for technical review. Would you be up for reviewing this one?

(Am still discussing science change with Graham Weedon at the Met Office. He agrees with my analysis so far, though has asked for some more plots/numbers to fully evidence up my assertion.)

alistairsellar · 2023-06-09T17:55:58Z

Have concluded science review discussion with Graham Weedon at MetO, having produced the additional plots/stats he suggested. See new doc for plots and my interpretation: soil_moisture_clims.pdf
Graham doesn't have a GitHub account so copying his conclusion:

Yes: I agree with your analysis. It seems to me that:
a) The consistent climatological patterns between old and new in all seasons, but the large scatter for JJA and SON of new v old values supports large improvements in the cci processing. Hence the transfer from the old to the new cci data will be more credible for assessing model output.
b) As I mentioned there is a much newer version available with further processing improvements (better cloud, ice and snow detection) and more reliable climatology from use of all satellite sensors and all satellite passes (allowing for different viewing angles) and more satellites data types processed.

bouweandela

Great to see these improvements @alistairsellar! From the code, it looks like provenance is not written for some of the output files. Could you add it for all files please? See https://docs.esmvaltool.org/en/latest/community/diagnostic.html#recording-provenance for more information.

esmvaltool/recipes/recipe_autoassess_landsurface_soilmoisture.yml

bouweandela · 2023-06-13T07:01:36Z

@esmvalbot Please run recipe_autoassess_landsurface_soilmoisture.yml

esmvalbot · 2023-06-13T07:01:40Z

Since @bouweandela asked, ESMValBot will run recipe recipe_autoassess_landsurface_soilmoisture.yml as soon as possible, output will be generated here

bouweandela · 2023-06-13T07:12:02Z

It looks like the recipe documentation page could also use some updates and formatting improvements. E.g. I see a bulleted list that is malformatted into a single line and mentions of the script autoassess_area_base.py which doesn't appear to be used by the recipe.

…oup/ESMValTool into autoassess_soilmoisture_obs_fly

alistairsellar · 2023-06-16T18:09:41Z

Provenance now sorted, for both the metrics files in first diag_script, and also the general AutoAssess metrics plotter. So I think that addresses all of @bouweandela's requests, apart from the citation warnings, which will be fixed by ESMValGroup/ESMValCore#2097.

@valeriupredoi ready for final check and merge, pending merge of ESMValGroup/ESMValCore#2097.

bouweandela · 2023-06-17T06:30:31Z

@esmvalbot Please run recipe_autoassess_landsurface_soilmoisture.yml

esmvalbot · 2023-06-17T06:30:35Z

Since @bouweandela asked, ESMValBot will run recipe recipe_autoassess_landsurface_soilmoisture.yml as soon as possible, output will be generated here

esmvalbot · 2023-06-17T06:36:43Z

ESMValBot is happy to report recipe recipe_autoassess_landsurface_soilmoisture.yml ran OK, output has been generated here

esmvaltool/diag_scripts/autoassess/land_surface_soilmoisture/soilmoisture.py

To keep pylint happy

bouweandela · 2023-06-19T10:15:46Z

@esmvalbot Please run recipe_autoassess_landsurface_soilmoisture.yml

esmvalbot · 2023-06-19T10:15:49Z

Since @bouweandela asked, ESMValBot will run recipe recipe_autoassess_landsurface_soilmoisture.yml as soon as possible, output will be generated here

esmvalbot · 2023-06-19T10:22:13Z

ESMValBot is happy to report recipe recipe_autoassess_landsurface_soilmoisture.yml ran OK, output has been generated here

esmvaltool/diag_scripts/autoassess/land_surface_soilmoisture/soilmoisture.py

alistairsellar · 2023-06-19T17:06:59Z

@esmvalbot Please run recipe_autoassess_landsurface_soilmoisture.yml

esmvalbot · 2023-06-19T17:07:03Z

Since @alistairsellar asked, ESMValBot will run recipe recipe_autoassess_landsurface_soilmoisture.yml as soon as possible, output will be generated here

esmvalbot · 2023-06-19T17:13:42Z

ESMValBot is happy to report recipe recipe_autoassess_landsurface_soilmoisture.yml ran OK, output has been generated here

valeriupredoi

very cool, guys, let's get this in 2.9 - many thanks for fixing this, a long standing issue on my to-do list 😁 🍺

valeriupredoi · 2023-06-20T08:56:20Z

esmvaltool/config-references.yml

+  rumbold_heather:
+    name: Heather, Rumbold
+    institute: Met Office, UK
+    orcid:


have you asked Heather? I can ask Steve on Whatsapp to ask her, if not 😁

According to ORCID website she does not have one. She is on maternity leave at present.

alistairsellar · 2023-06-20T13:26:48Z

Thanks for your help @bouweandela and @valeriupredoi!

…old_and_clone_task_rtw * recipe_test_workflow_prototype: (199 commits) #3169: Upgrade the RTW to work with ESMValTool v2.8.0 [Condalock] Update Linux condalock file (#3237) Modified links to the tutorial (#3236) Add ESMValCore release `v2.8.1` into the documentation (#3235) Generate climatology on the fly for AutoAssess soil moisture (#3197) New recipe and diagnostic for Arctic-midlatitude research (#3021) Fixed pandas diagnostics for pandas>=2.0.0 (#3209) Update obs4MIPs dataset to the current naming scheme in recipe_smpi.yml (#2991) Add variable long names to provenance record in monitoring diagnostics (#3222) Extension of NASA MERRA2 CMORizer (cl, cli, clivi, clw, clwvi) (#3167) Remove "fx_variable" from recipe_wenzel14jgr.yml (#3212) [Condalock] Update Linux condalock file (#3217) Add Seaborn diagnostic (#3155) Remove fx_variables from ipccwg1ar5ch9 recipes (#3215) Remove "fx_variable" from recipe_tebaldi21esd.yml (#3211) Update recipe_impact.yml to work with newer versions of `pandas` (#3220) Use ESMValCore v2.9.0 release candidates (#3219) [Github Actions ] Check if python minor version changed after Julia install in development installation test (#3213) New plot_type 1d_profile in monitor (#3178) Add support for using a dask distributed scheduler (#3151) ...

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

alistairsellar added 7 commits December 18, 2022 21:22

Add ORCID

a982577

Remove reference to climatology files

fb5cb14

Try adding CCI obs

874bbb3

Make CCI additional_dataset and add preprocessor

413311a

New function to write metrics files

04e7d3e

Fix provenance

e93d2a3

Merge branch 'main' into autoassess_soilmoisture_obs_fly

1a2aca3

alistairsellar requested a review from valeriupredoi May 26, 2023 19:03

alistairsellar added 6 commits June 2, 2023 11:21

Fix the easy Codacy errors

adb36fd

Last one requires a minor refactor

Break provenance into own function for pylint

8ff4a01

Fixes problem of too-many-locals (Alternative approach would be to find another pub?)

Fix new style error

5ab7e63

Remove doc mention of autoassess_area_base...

686ea3e

...since diag_script is called directly

Remove from broken recipes list

2efbc26

Merge branch 'main' into autoassess_soilmoisture_obs_fly

2f7e6d3

alistairsellar added this to the v2.9.0 milestone Jun 12, 2023

alistairsellar self-assigned this Jun 12, 2023

alistairsellar added looking for technical reviewer AutoAssess Issues relevant to the conversion of AutoAssess metrics from Met Office to ESMValTool labels Jun 12, 2023

bouweandela requested changes Jun 13, 2023

View reviewed changes

esmvaltool/recipes/recipe_autoassess_landsurface_soilmoisture.yml Outdated Show resolved Hide resolved

bouweandela removed the looking for technical reviewer label Jun 13, 2023

alistairsellar added 2 commits June 16, 2023 18:23

Merge branch 'main' into autoassess_soilmoisture_obs_fly

1aa65b1

Merge branch 'autoassess_soilmoisture_obs_fly' of github.com:ESMValGr…

c2c6104

…oup/ESMValTool into autoassess_soilmoisture_obs_fly

bouweandela reviewed Jun 19, 2023

View reviewed changes

esmvaltool/diag_scripts/autoassess/land_surface_soilmoisture/soilmoisture.py Outdated Show resolved Hide resolved

alistairsellar added 4 commits June 19, 2023 10:24

Add provenance for intermediate cube save

473af6d

Move unit conversion into its own function

e8cebfe

To keep pylint happy

Merge branch 'main' into autoassess_soilmoisture_obs_fly

64225ed

Fix style error

e64ee64

bouweandela reviewed Jun 19, 2023

View reviewed changes

esmvaltool/diag_scripts/autoassess/land_surface_soilmoisture/soilmoisture.py Outdated Show resolved Hide resolved

bouweandela reviewed Jun 19, 2023

View reviewed changes

esmvaltool/diag_scripts/autoassess/land_surface_soilmoisture/soilmoisture.py Outdated Show resolved Hide resolved

Make provenance specific to model dataset

10cb722

bouweandela approved these changes Jun 20, 2023

View reviewed changes

valeriupredoi added diagnostic enhancement labels Jun 20, 2023

valeriupredoi approved these changes Jun 20, 2023

View reviewed changes

bouweandela merged commit be55fec into main Jun 20, 2023
6 checks passed

bouweandela deleted the autoassess_soilmoisture_obs_fly branch June 20, 2023 11:48

remi-kazeroni mentioned this pull request Jun 20, 2023

Recipes with issues discovered during the 2.8.0 release runs #3140

Closed

jvegreg pushed a commit that referenced this pull request Jan 14, 2024

Generate climatology on the fly for AutoAssess soil moisture (#3197)

90f4a61

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate climatology on the fly for AutoAssess soil moisture #3197

Generate climatology on the fly for AutoAssess soil moisture #3197

alistairsellar commented May 26, 2023 •

edited

alistairsellar commented May 26, 2023

alistairsellar commented May 26, 2023

alistairsellar commented May 26, 2023

alistairsellar commented May 26, 2023

alistairsellar commented Jun 2, 2023

alistairsellar commented Jun 9, 2023

bouweandela left a comment

bouweandela commented Jun 13, 2023

esmvalbot bot commented Jun 13, 2023

bouweandela commented Jun 13, 2023

alistairsellar commented Jun 16, 2023

bouweandela commented Jun 17, 2023

esmvalbot bot commented Jun 17, 2023

esmvalbot bot commented Jun 17, 2023

bouweandela commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

alistairsellar commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

valeriupredoi left a comment

valeriupredoi Jun 20, 2023

alistairsellar Jun 20, 2023

alistairsellar commented Jun 20, 2023

Generate climatology on the fly for AutoAssess soil moisture #3197

Generate climatology on the fly for AutoAssess soil moisture #3197

Conversation

alistairsellar commented May 26, 2023 • edited

Description

Before you get started

Checklist

New or updated recipe/diagnostic

alistairsellar commented May 26, 2023

alistairsellar commented May 26, 2023

Science impact

alistairsellar commented May 26, 2023

alistairsellar commented May 26, 2023

alistairsellar commented Jun 2, 2023

alistairsellar commented Jun 9, 2023

bouweandela left a comment

Choose a reason for hiding this comment

bouweandela commented Jun 13, 2023

esmvalbot bot commented Jun 13, 2023

bouweandela commented Jun 13, 2023

alistairsellar commented Jun 16, 2023

bouweandela commented Jun 17, 2023

esmvalbot bot commented Jun 17, 2023

esmvalbot bot commented Jun 17, 2023

bouweandela commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

alistairsellar commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

esmvalbot bot commented Jun 19, 2023

valeriupredoi left a comment

Choose a reason for hiding this comment

valeriupredoi Jun 20, 2023

Choose a reason for hiding this comment

alistairsellar Jun 20, 2023

Choose a reason for hiding this comment

alistairsellar commented Jun 20, 2023

alistairsellar commented May 26, 2023 •

edited