Implement the climwip weighting scheme in a recipe and diagnostic

[JaroCamphuijsen]

Before you start, read CONTRIBUTING.md and the guide for diagnostic developers.

Please discuss your idea with the development team before getting started, to avoid disappointment later. The way to do this is to open a new issue on GitHub. If you are planning to modify an existing functionality, please discuss it with the original author(s) by tagging them in the issue.

---

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• Give this pull request a descriptive title that can be used as a one line summary in a changelog
• Make sure your code is composed of functions of no more than 50 lines and uses meaningful names for variables
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes
• (Only if really necessary) Add any additional dependencies needed for the diagnostic script to setup.py, esmvaltool/install/R/r_requirements.txt or esmvaltool/install/Julia/julia_requirements.txt (depending on the language of your script) and also to package/meta.yaml for conda dependencies (includes Python and others, but not R/Julia)
• If new dependencies are introduced, check that the license is compatible with Apache2.0

New recipe/diagnostic

• Add documentation for the recipe to the doc/sphinx/source/recipes folder and add a new entry to index.rst
• Add provenance information

---

Closes #1640

[lukasbrunner]

I do no longer think that the sea masking is the issue. It seems to be the detrended_std operation.

ESMValTool/esmvaltool/recipes/recipe_climwip.yml

Lines 63 to 69 in 10b25f1

	detrended_std:
	<<: *general
	detrend:
	dimension: time
	method: linear
	climate_statistics:
	operator: std_dev

If I change this line also to climatological_mean everything works fine:

ESMValTool/esmvaltool/recipes/recipe_climwip.yml

Line 95 in 10b25f1

preprocessor: detrended_std

For this case (including my suggestions above, I get very similar results when comparing the output from the ESMValTool recipe and our implementation:

It would still be interesting to understand where the difference comes from, but this is a detail I'm okay with postponing.

[ruthlorenz]

I do no longer think that the sea masking is the issue. It seems to be the detrended_std operation.

I agree that the masking works fine for the climatology, so detrended_std is causing the issue. Somehow the detrend preprocessor messes with the masking (as already reported here ESMValGroup/ESMValCore#768).

But these are problems with the preprocessor functions, not the weighting code as such.
I suggest to sort out a more realistic example recipe at a later stage and change the recipe as suggested by Lukas. It is minimal (and maybe scientifically not great) but works.

[Peter9192]

@lukasbrunner In the last commit I used YAML anchors to condense the recipe a bit. Can you check if you prefer this or not? Otherwise I will revert it.

[lukasbrunner]

@lukasbrunner In the last commit I used YAML anchors to condense the recipe a bit. Can you check if you prefer this or not? Otherwise I will revert it.

I think this is quite neat feel free to leave it like that from my side!

[ruthlorenz]

looks good to me too, I like to see the variable groups as examples.

if we really wanted to include detrend_std we could do like this:

  detrended_std:
    custom_order: true
    <<: *general
    annual_statistics:
      operator: mean
    detrend:
      dimension: time
      method: linear
    climate_statistics:
      operator: std_dev
    mask_landsea:
      mask_out: sea

reapplying the ocean mask at the end. but its a bit of a hack and the masking is different because of the different order of operations...

[ruthlorenz]

Looks good, works, and will be even better when we figured out the issues with the preprocessors ;-)

[Peter9192]

@ESMValGroup/esmvaltool-coreteam this recipe has a technical approval from @bouweandela and a scientific approval from @ruthlorenz . Can this be merged now? 1 small note is that 'natsort' was added as a dependency after Bouwe's review, but it this looks like a decent package, so I think it should not be a problem.

[bouweandela]

The natsort package would also need to be added to package/meta.yml, or the conda package will be broken.