Add mapplot diagnostic to ClimWIP #1864

lukasbrunner · 2020-10-15T13:19:13Z

Tasks

Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
Give this pull request a descriptive title that can be used as a one line summary in a changelog
Make sure your code is composed of functions of no more than 50 lines and uses meaningful names for variables
Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
Please use yamllint to check that your YAML files do not contain mistakes

Modified recipe/diagnostic

Update documentation for the recipe to the doc/sphinx/source/recipes folder
Assign the author(s) of the affected recipe(s) as reviewer(s)

Closes #1863

lukasbrunner · 2020-10-15T13:21:59Z

Example output:

Peter9192

Hey Lukas!
Nice work on implementing this and passing all checks (codacy is okay)! I haven't tested it yet, but I had a quick look at the code. I left a few comments/ideas, hope to have a closer look at it later.

One small side note is that the figure headers weren't immediately obvious to me, I think because the second one has two 'minuses'. perhaps you could call it '(weighted minus unweighted) temperature anomalies' ?

esmvaltool/diag_scripts/weighting/weighted_temperature_graph.py

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

esmvaltool/recipes/recipe_climwip.yml

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

doc/sphinx/source/recipes/recipe_climwip.rst

lukasbrunner · 2020-10-15T16:11:14Z

Thanks for the comments @Peter9192! I agree that a shared utils file could make sense, in fact I started from your plotting script so a lot of the basic functions are identical. I will start to restructure this.

About the anomalies preprocessor: can you write down the syntax for this? I failed and @ruthlorenz also did not know how to do that. You use it for the timeseries plot but there the reference period is in the period you then also plot. I need to calculate two climatologies in non-overlapping time periods and subtract them from each other.

Peter9192 · 2020-10-16T07:23:07Z

About the anomalies preprocessor: can you write down the syntax for this?

Not tested, but I think it should be something like:

preprocessors:
  tas_anomalies:
    custom_order: true
    regrid ... # with appropriate keywords
    anomalies: 
      period: full  # this means substracting the overall mean of period below.
      reference:  # this is your reference period, i.e. the climate you want to substract from the other one
        start_year: 1981
        start_month: 1
        start_day: 1
        end_year: 2010
        end_month: 12
        end_day: 31
      standardize: false  # whether you want the result in K or in % (or decimal, not sure what it gives).
    # So far you now still have the full period, but only relative to this reference period. So now you can continue:
    extract_time: 
        start year: 2081
        start .... # etc.
    climate_statistics:
      operator: mean
    

diagnostics:
  variables:
    tas_map:
      short_name: tas
      preprocessor: tas_anomalies
      start_year: 1981  # this can be defined here, or in the datasets section. 
      end_year: 2100  # But this span needs to encompass both reference and target period.

So to summarize:

average over reference period
substract that average from all other times (all this is done by the anomalies preprocessor)
(extract and) average over the target period

I think you're description is swapping steps 2 and 3, but I don't think that makes a difference.
Does that help?

bouweandela · 2020-10-16T11:45:04Z

The preprocessor above may need ESMValGroup/ESMValCore#796 to work.

Peter9192 · 2020-10-16T13:41:24Z

The preprocessor above may need ESMValGroup/ESMValCore#796 to work.

Good point. So maybe the last 2 steps can be done in the diagnostic script until then.

ruthlorenz

Looks good to me in general, a few suggestions in my comments.
The only thing that really needs resolving is the set up of the recipe and if the required python version can be upgraded easily.

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

doc/sphinx/source/recipes/recipe_climwip.rst

esmvaltool/recipes/recipe_climwip.yml

lukasbrunner · 2020-10-21T14:21:09Z

I've fixwd both issues @ruthlorenz. The assignment expressions were not really necessary and without them it should be okay to run under 3.6.

lukasbrunner · 2020-10-23T07:46:22Z

@stefsmeets @Peter9192 could one of you have a quick look at this again and authorize the merge (if everything looks good)?

Peter9192 · 2020-10-23T08:00:49Z

Yes, I absolutely want to get this merged! I'll try to get to it today, but no promises...

Peter9192

Hey @lukasbrunner very nice work. The code works, and already look very good. I have a couple of suggestions. Could you have a look at them?

Please also have a look at the contributing guidelines. We try to format the code consistently, by using yapf and isort. They are both quite easy to use. If you want to integrate all checks and auto-formatting, you can use pre-commit. It's quite useful once you get used to it.

esmvaltool/diag_scripts/weighting/plot_utilities.py

esmvaltool/diag_scripts/weighting/weighted_temperature_graph.py

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

Co-authored-by: Peter Kalverla <peter.kalverla@gmx.com>

lukasbrunner · 2020-10-28T16:50:48Z

Slowly getting there :) This might be the wrong place for this discussion but since @bouweandela raised it: It does not seem to make sense to import xarray just to use in an annotation? What is the preferred solution here?

undefined name 'xr' (F821)
def visualize_and_save_temperature(temperature: 'xr.DataArray', cfg: dict,

Update: okay now I added it and it complains that it is unused...

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

stefsmeets · 2020-10-29T08:13:37Z

Slowly getting there :) This might be the wrong place for this discussion but since @bouweandela raised it: It does not seem to make sense to import xarray just to use in an annotation? What is the preferred solution here?

undefined name 'xr' (F821)
def visualize_and_save_temperature(temperature: 'xr.DataArray', cfg: dict,

Update: okay now I added it and it complains that it is unused...

We had this discussion before when developing the recipe. I'm a big fan of type annotations and use them whenever I can. There is nothing wrong with the code here. It is just that flake8 and CircleCI do not like it one way or the other and (even worse) do not agree with each other what is right. I think in the end we had to use xarray somewhere, so we could get around the 'unused variable' issue. If you cannot find a way to use xarray somewhere, best to avoid the issue altogether and remove the annotation to keep the CI/flake8 happy.

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

bouweandela · 2020-10-29T10:22:12Z

This might be the wrong place for this discussion but since @bouweandela raised it: It does not seem to make sense to import xarray just to use in an annotation? What is the preferred solution here?

I think this is a very good place to ask this question. The solution of not using type annotations is absolutely fine, but if you want to use them, you need to import it, because otherwise there is no way of telling what you mean with xr. This is a bug in pylint, reported in several issues, e.g. this one: pylint-dev/pylint#3299.

bouweandela · 2020-10-29T10:22:55Z

@esmvalbot recipe_climwip.yml

esmvalbot · 2020-10-29T10:22:58Z

Starting recipe recipe_climwip.yml, output will be generated here

esmvalbot · 2020-10-29T10:35:40Z

Bot failed to run recipe recipe_climwip.yml: exit is 1, output has been generated here

Peter9192 · 2020-10-29T10:44:48Z

Bot failed to run recipe recipe_climwip.yml: exit is 1, output has been generated here

@bouweandela I like the test case for the new bot, but let's try to solve that issue somewhere else. I tested the recipe locally during the technical review, and without the missing data the recipe runs fine. Let's get this merged so Lukas can continue with follow-up PRs.

bouweandela · 2020-10-29T15:29:20Z

@esmvalbot recipe_climwip.yml

esmvalbot · 2020-10-29T15:29:23Z

Starting recipe recipe_climwip.yml, output will be generated here

esmvalbot · 2020-10-29T15:43:38Z

Bot failed to run recipe recipe_climwip.yml: exit is 1, output has been generated here

bouweandela · 2020-10-29T15:55:51Z

Ok, I changed the configuration of the bot so it now finds all the data, but now it's failing on something that looks related to the cartopy version, SciTools/cartopy#1615. I'll have a look.

Peter9192 · 2020-10-30T08:15:33Z

Ok, I changed the configuration of the bot so it now finds all the data, but now it's failing on something that looks related to the cartopy version, SciTools/cartopy#1615. I'll have a look.

That's the same problem we had with the example recipe in #1882.

Peter9192 · 2020-11-03T13:47:25Z

@bouweandela can you please merge this? Let's solve the bot issue somewhere else (e.g. #1882) and not block Lukas' work on it.

bouweandela · 2020-11-04T10:28:12Z

Let's have a final try to get it working, the current failure is not related to the bot, but to the wrong versions of dependencies, so at the moment this recipe will not work for anyone who does a new installation. I think this can be fixed by #1898. Once we have that merged into master, we can merge master into this branch and have a final go at running it with the bot.

…ip_recipe_map

bouweandela · 2020-11-04T11:27:11Z

@esmvalbot recipe_climwip.yml

esmvalbot · 2020-11-04T11:27:13Z

Starting recipe recipe_climwip.yml, output will be generated here

esmvalbot · 2020-11-04T11:51:29Z

Recipe recipe_climwip.yml ran OK, output has been generated here

Lukas Brunner added 2 commits October 13, 2020 10:13

Add first try of a mapplot recipe

146c4c1

Add mapplot diagnostic, recipe part, and docu

7735633

Resolve some Codacy issues

83b331e

lukasbrunner self-assigned this Oct 15, 2020

lukasbrunner added the EUCP www.eucp-project.eu label Oct 15, 2020

lukasbrunner requested review from ruthlorenz, Peter9192 and stefsmeets October 15, 2020 13:30

lukasbrunner marked this pull request as ready for review October 15, 2020 13:31

Peter9192 reviewed Oct 15, 2020

View reviewed changes

Lukas Brunner added 5 commits October 16, 2020 17:41

Create and move some functions

9d273b3

Move some functions to utils

b8de236

Move some functions to utils; remove code duplication

2b72e51

Add example figure

5a0f066

Fix codacy isses

bcd70e6

ruthlorenz reviewed Oct 21, 2020

View reviewed changes

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py Outdated Show resolved Hide resolved

doc/sphinx/source/recipes/recipe_climwip.rst Show resolved Hide resolved

esmvaltool/recipes/recipe_climwip.yml Show resolved Hide resolved

Lukas Brunner added 2 commits October 21, 2020 16:04

Correct filename

fc7ce71

Add Python 3.6 compitability

366116d

lukasbrunner requested a review from Peter9192 October 23, 2020 07:52

Peter9192 requested changes Oct 23, 2020

View reviewed changes

lukasbrunner and others added 2 commits October 23, 2020 16:19

Update esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

a415eae

Co-authored-by: Peter Kalverla <peter.kalverla@gmx.com>

Update esmvaltool/diag_scripts/weighting/weighted_temperature_map.py

47d880f

Co-authored-by: Peter Kalverla <peter.kalverla@gmx.com>

Lukas Brunner added 2 commits October 28, 2020 18:10

Resolve Codacy issues

44ce352

Fix bug with linebreaks

6123980

lukasbrunner commented Oct 28, 2020

View reviewed changes

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py Outdated Show resolved Hide resolved

Peter9192 reviewed Oct 29, 2020

View reviewed changes

esmvaltool/diag_scripts/weighting/weighted_temperature_map.py Outdated Show resolved Hide resolved

Clean up annotation issue

c0ae47c

stefsmeets approved these changes Oct 30, 2020

View reviewed changes

Peter9192 mentioned this pull request Nov 3, 2020

Add the option to weight variable groups in ClimWIP #1856

Merged

9 tasks

Merge branch 'master' of github.com:ESMValGroup/ESMValTool into climw…

092e01b

…ip_recipe_map

bouweandela merged commit ccd1eec into master Nov 4, 2020

bouweandela deleted the climwip_recipe_map branch November 4, 2020 13:04

bouweandela mentioned this pull request Nov 5, 2020

Monthly ESMValtool meeting November #1836

Closed

schlunma mentioned this pull request Nov 10, 2020

Added clip_start_end_year preprocessor ESMValGroup/ESMValCore#796

Merged

9 tasks

jvegreg added the diagnostic label Feb 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mapplot diagnostic to ClimWIP #1864

Add mapplot diagnostic to ClimWIP #1864

lukasbrunner commented Oct 15, 2020 •

edited

Loading

lukasbrunner commented Oct 15, 2020

Peter9192 left a comment

lukasbrunner commented Oct 15, 2020

Peter9192 commented Oct 16, 2020

bouweandela commented Oct 16, 2020

Peter9192 commented Oct 16, 2020

ruthlorenz left a comment •

edited

Loading

lukasbrunner commented Oct 21, 2020

lukasbrunner commented Oct 23, 2020

Peter9192 commented Oct 23, 2020

Peter9192 left a comment

lukasbrunner commented Oct 28, 2020 •

edited

Loading

stefsmeets commented Oct 29, 2020

bouweandela commented Oct 29, 2020

bouweandela commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

Peter9192 commented Oct 29, 2020

bouweandela commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

bouweandela commented Oct 29, 2020 •

edited

Loading

Peter9192 commented Oct 30, 2020

Peter9192 commented Nov 3, 2020

bouweandela commented Nov 4, 2020

bouweandela commented Nov 4, 2020

esmvalbot bot commented Nov 4, 2020

esmvalbot bot commented Nov 4, 2020

Add mapplot diagnostic to ClimWIP #1864

Add mapplot diagnostic to ClimWIP #1864

Conversation

lukasbrunner commented Oct 15, 2020 • edited Loading

lukasbrunner commented Oct 15, 2020

Peter9192 left a comment

Choose a reason for hiding this comment

lukasbrunner commented Oct 15, 2020

Peter9192 commented Oct 16, 2020

bouweandela commented Oct 16, 2020

Peter9192 commented Oct 16, 2020

ruthlorenz left a comment • edited Loading

Choose a reason for hiding this comment

lukasbrunner commented Oct 21, 2020

lukasbrunner commented Oct 23, 2020

Peter9192 commented Oct 23, 2020

Peter9192 left a comment

Choose a reason for hiding this comment

lukasbrunner commented Oct 28, 2020 • edited Loading

stefsmeets commented Oct 29, 2020

bouweandela commented Oct 29, 2020

bouweandela commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

Peter9192 commented Oct 29, 2020

bouweandela commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

esmvalbot bot commented Oct 29, 2020

bouweandela commented Oct 29, 2020 • edited Loading

Peter9192 commented Oct 30, 2020

Peter9192 commented Nov 3, 2020

bouweandela commented Nov 4, 2020

bouweandela commented Nov 4, 2020

esmvalbot bot commented Nov 4, 2020

esmvalbot bot commented Nov 4, 2020

lukasbrunner commented Oct 15, 2020 •

edited

Loading

ruthlorenz left a comment •

edited

Loading

lukasbrunner commented Oct 28, 2020 •

edited

Loading

bouweandela commented Oct 29, 2020 •

edited

Loading