Implement shape parameter calibration for ClimWIP #1905

lukasbrunner · 2020-11-24T12:46:56Z

Tasks

Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
Give this pull request a descriptive title that can be used as a one line summary in a changelog
Make sure your code is composed of functions of no more than 50 lines and uses meaningful names for variables
Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
Please use yamllint to check that your YAML files do not contain mistakes

Modified recipe/diagnostic

Update documentation for the recipe to the doc/sphinx/source/recipes folder
Update provenance information if needed
Assign the author(s) of the affected recipe(s) as reviewer(s)

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.

Closes #1902
Documentation: https://esmvaltool--1905.org.readthedocs.build/en/1905/recipes/recipe_climwip.html

lukasbrunner · 2020-11-24T20:18:19Z

Hey @Peter9192 ! I now have a first running version. If you have time tomorrow at some point we could discuss it. It is quite similar to my pseudo code idea with the addition of the scipy.minimize

Peter9192 · 2020-11-25T10:48:56Z

Hey @lukasbrunner I'm on it now. I totally see why you made the utilities file, but it's quite hard to compare the changes now. Do you think you could split up the commits a bit more logically (maybe move everything first without changes, then the actual changes)? That would make it much easier to go commit by commit and evaluate the relevant changes only.

lukasbrunner · 2020-11-25T11:03:33Z

Ah sorry yes! I always get ahead of myself and then there are no atomic commits any more... but really the changes to climwip.py are limited to moving functions to utilities.py and utilities.py contains only functions from the original climwip.py. So the only real changes are isolated in calibrate_sigmas.py (except for the call in climwip.py and a very small piece cross-checking if sigmas are already set)

Do you still want me to redo this? Might be a good learning process for me anyway so that would be okay :)

Peter9192 · 2020-11-25T11:16:54Z

Let's discuss it first

esmvaltool/diag_scripts/weighting/calibrate_sigmas.py

Peter9192 · 2020-11-25T17:10:41Z

esmvaltool/diag_scripts/weighting/calibrate_sigmas.py

+    if confident:
+        return performance_sigma
+    return 9999 - performance_sigma
+


Apologies for cross posting 😋

Maybe this is still too simple, but I wanted to offer a thought. If we specify confidence as a spectrum around 0.80, where anything below 0.80 is overconfident (and lower is worse), and anything above 0.80 as 'underconfident' (diffident? ; higher is worse), then minimizing something like abs(confidence - target_confidence) where target_confidence=0.80 could maybe work?

- calculate_model_distances: update docstring - calculate_model_distances: add argument to name new dimension name - compute_overall_mean: simplify median calculation - combine_ensemble_members: update docstring - combine_ensemble_members: allow passing of dimension names to be combined - calculate_weights: move actualy calculation to a new function to get more flexible. This function becomes a wrapper doing xarray stuff

lukasbrunner · 2020-12-02T17:07:14Z

Hey @Peter9192 !
My next try with what I think is the most basic version of the sigma calibration :) I'm aware that its not quite polished now and the code still contains a couple of open questions.

I now use brute instead of minimize. I realize that this might be slightly less elegant but it is more practicable because it allows a plot of what is going on. I think this is quite important to evaluate if the perfect model test did work. In addition it makes the exact definition of the cost function less important.
Not sure if my method of extracting the values from the parallel loop inside brute using dict is quite how this should be done...
I realize the plot is a bit busy and not easy to interpret but I tried to include all the relevant information and I can add a description to the docs
I added a new recipe for running the sigma calibration *_test_performance_sigma.yml (the old one did not have enough model for meaningful results) and renamed the old one into *_test_basic.yml. Not sure if this is something that is used in ESMValTool but in the end I think it needs to be clear that both of them are exactly that: toy examples with out much real meaning. Once Ruth is done we can add her recipe as 'real' application.

Example figure sigma calibration

Figure: Ratio of perfect models inside of the weighted 80% range (thick black line) depending on the performance sigma value. The smallest not-overconfident sigma is marked by the dashed vertical line. Ratio of weighted/unweighted 80% ranges (min, mean, max; gray line and shading) depending on the performance sigma value. Cost function for finding the ideal sigma value (red, right axis).

Additional information:
I also show the reference we try to reach (dashed horizontal line) and the unweighted baseline (doted horizontal line). The difference between the two is that for the baseline the perfect model is excluded from the distribution meaning baseline <= reference and baseline = reference for number of models -> infinity. In addition the ratio of perfect models -> unweighted baseline for sigma -> infinity

I also show a measure of "sharpness", i.e., the ratio between weighted and unweighted spread for each perfect model. The smaller the value the stronger the constraint of the weighting, with sharpness -> 1 for sigma -> infinity.

The cost function needs to basic properties: If no confident sigma can be found it needs to pick the largest possible value (weakest weighting) and for all confident values it needs to pick the smallest possible value.

esmvalbot · 2021-02-04T17:15:27Z

Since @Peter9192 asked, ESMValBot will run recipe recipe_climwip_test_performance_sigma.yml as soon as possible, output will be generated here

esmvalbot · 2021-02-04T17:35:18Z

ESMValBot is sorry to report it failed to run recipe recipe_climwip_test_performance_sigma.yml: exit is 1, output has been generated here

Peter9192 · 2021-02-09T08:41:55Z

@ruthlorenz would you have time to do the scientific review this week? If you can try if it runs for you on the ETH resources, then I can stop trying with the bot..

ruthlorenz · 2021-02-10T13:39:25Z

yes I can do a scientific review

ruthlorenz

I added a few little things that need changing. Otherwise looks good to me and runs fine.
However, I am not completely independent since I run on the same servers as @lukasbrunner.

doc/sphinx/source/recipes/recipe_climwip.rst

esmvaltool/diag_scripts/weighting/climwip/calibrate_sigmas.py

esmvaltool/diag_scripts/weighting/climwip/core_functions.py

doc/sphinx/source/recipes/recipe_climwip.rst

esmvaltool/recipes/recipe_climwip_test_performance_sigma.yml

…Group/ESMValTool into climwip_sigma_calibration

…ration

lukasbrunner · 2021-02-15T09:40:25Z

Thanks @ruthlorenz for the careful check!!
@Peter9192 not sure why CI fails now I only touched comments since my earlier commits. (except maybe the added information to the recipe?) Or the merge of the master?

Peter9192 · 2021-02-15T14:18:53Z

Seems unrelated to the PR, something with the test_cmorize_obs. Guessing this might be due to some of the refactoring of the logger we did recently in the core (@stefsmeets ?).

stefsmeets · 2021-02-15T14:44:29Z

Seems unrelated to the PR, something with the test_cmorize_obs. Guessing this might be due to some of the refactoring of the logger we did recently in the core (@stefsmeets ?).

I don't know what happened with this, should have been fixed by #2020

valeriupredoi · 2021-02-15T15:15:31Z

#2020 fixed it for the current development master branch of ESMValCore, but broke it for the released version which is used for the CI tests stirring the wrath of @bouweandela while at it 😁

Peter9192 · 2021-02-15T16:01:00Z

One more minor technical remark @lukasbrunner, can you run yamllint on the changed recipes? I think you might want to trim down the line length on some of the descriptions you've added.

Peter9192 · 2021-02-15T17:01:48Z

Thanks!
Since the CI issue has nothing to do with this PR I'm happy to see this get merged

@ESMValGroup/esmvaltool-coreteam

bouweandela · 2021-02-16T08:37:45Z

Could you please merge the master branch into this pull request, so all the tests pass?

…ration

valeriupredoi · 2021-02-17T14:39:13Z

I take it that @ruthlorenz ok-ed this scientologically and @Peter9192 technically right? :grin: I am going to change labels and pending test passing I am going to merge, please object if otherwise :+1:

ruthlorenz

All good now

Add pseudo code skeleton to start with

9af90fe

lukasbrunner added diagnostic EUCP www.eucp-project.eu labels Nov 24, 2020

lukasbrunner self-assigned this Nov 24, 2020

lukasbrunner changed the title ~~Implement on the fly shape parameter claibration~~ Implement shape parameter claibration for ClimWIP Nov 24, 2020

Lukas Brunner added 3 commits November 24, 2020 20:48

Mode functions to utilities & add option to calibrate sigma on the fly

7200745

Move function to utilities

2bd7640

Fix but, add checks

a9901aa

Peter9192 reviewed Nov 25, 2020

View reviewed changes

esmvaltool/diag_scripts/weighting/calibrate_sigmas.py Outdated Show resolved Hide resolved

Peter9192 reviewed Nov 25, 2020

View reviewed changes

esmvaltool/diag_scripts/weighting/calibrate_sigmas.py Outdated Show resolved Hide resolved

Peter9192 reviewed Nov 25, 2020

View reviewed changes

Lukas Brunner added 12 commits November 26, 2020 14:26

Merge branch 'master' into climwip_sigma_calibration

9bfd91d

Remove outdated file

f6485e0

Remove outdated file

fd1d859

Delete outdated file

dd98f5f

Merge original file from master

5e86b52

Add on the fly performance sigma calibration

c1de270

Update some functions to be more general

74e7093

Update call of combine_ensemble_members

7e7fd45

Fix bug

3888c8d

Add recipe to test performance sigma calibration

7c66fb2

Add minimal version of performance sigma calibration

b2edb64

stefsmeets changed the title ~~Implement shape parameter claibration for ClimWIP~~ Implement shape parameter calibration for ClimWIP Dec 3, 2020

stefsmeets self-requested a review December 3, 2020 09:40

bouweandela added approved by technical reviewer looking for scientific reviewer labels Feb 9, 2021

ruthlorenz requested changes Feb 11, 2021

View reviewed changes

Lukas Brunner added 5 commits February 15, 2021 10:08

Add some description

b76aa17

Merge branch 'climwip_sigma_calibration' of https://github.com/ESMVal…

4b637e8

…Group/ESMValTool into climwip_sigma_calibration

Merge remote-tracking branch 'origin/master' into climwip_sigma_calib…

9a19481

…ration

Fix spelling mistakes in documentation

1f25285

Insert linebreak

19b45c0

Fix line length

c001ba9

lukasbrunner requested a review from ruthlorenz February 16, 2021 07:58

Lukas Brunner and others added 2 commits February 16, 2021 09:43

Merge remote-tracking branch 'origin/master' into climwip_sigma_calib…

dd7c084

…ration

Merge branch 'master' into climwip_sigma_calibration

19ebe52

valeriupredoi added approved by scientific reviewer and removed looking for scientific reviewer labels Feb 17, 2021

ruthlorenz approved these changes Feb 17, 2021

View reviewed changes

Merge branch 'master' into climwip_sigma_calibration

4c7bd15

valeriupredoi merged commit 1e5b5dd into master Feb 17, 2021

valeriupredoi deleted the climwip_sigma_calibration branch February 17, 2021 17:43

Implement shape parameter calibration for ClimWIP #1905

Implement shape parameter calibration for ClimWIP #1905

lukasbrunner commented Nov 24, 2020 •

edited by Peter9192

lukasbrunner commented Nov 24, 2020

Peter9192 commented Nov 25, 2020

lukasbrunner commented Nov 25, 2020 •

edited

Peter9192 commented Nov 25, 2020

Peter9192 Nov 25, 2020

lukasbrunner commented Dec 2, 2020

esmvalbot bot commented Feb 4, 2021

esmvalbot bot commented Feb 4, 2021

Peter9192 commented Feb 9, 2021

ruthlorenz commented Feb 10, 2021

ruthlorenz left a comment

lukasbrunner commented Feb 15, 2021

Peter9192 commented Feb 15, 2021

stefsmeets commented Feb 15, 2021

valeriupredoi commented Feb 15, 2021

Peter9192 commented Feb 15, 2021

Peter9192 commented Feb 15, 2021 •

edited

bouweandela commented Feb 16, 2021

valeriupredoi commented Feb 17, 2021

ruthlorenz left a comment

Implement shape parameter calibration for ClimWIP #1905

Implement shape parameter calibration for ClimWIP #1905

Conversation

lukasbrunner commented Nov 24, 2020 • edited by Peter9192

lukasbrunner commented Nov 24, 2020

Peter9192 commented Nov 25, 2020

lukasbrunner commented Nov 25, 2020 • edited

Peter9192 commented Nov 25, 2020

Peter9192 Nov 25, 2020

Choose a reason for hiding this comment

lukasbrunner commented Dec 2, 2020

esmvalbot bot commented Feb 4, 2021

esmvalbot bot commented Feb 4, 2021

Peter9192 commented Feb 9, 2021

ruthlorenz commented Feb 10, 2021

ruthlorenz left a comment

Choose a reason for hiding this comment

lukasbrunner commented Feb 15, 2021

Peter9192 commented Feb 15, 2021

stefsmeets commented Feb 15, 2021

valeriupredoi commented Feb 15, 2021

Peter9192 commented Feb 15, 2021

Peter9192 commented Feb 15, 2021 • edited

bouweandela commented Feb 16, 2021

valeriupredoi commented Feb 17, 2021

ruthlorenz left a comment

Choose a reason for hiding this comment

lukasbrunner commented Nov 24, 2020 •

edited by Peter9192

lukasbrunner commented Nov 25, 2020 •

edited

Peter9192 commented Feb 15, 2021 •

edited