Recipe to plot generic output from the preprocessor #2184

jvegreg · 2021-05-29T09:41:13Z

This recipe introduce a diagnostic that can plot in a configurable way some direct outputs from the preprocessor. A version of this recipe is what we are using at BSC to monitor our model runs but can be used also for exploratory analysis of already run experiments.

Still missing from this recipe, to be fixed before merge

Provenance recording
Documentation
Release the next version of MapGenerator to solve some graphic issues

Some comments:

The diagnostic is meant to grow in the future with new types of plots
Plots are stored in a central location because this recipe is mean to be run multiple times (because the experiment is still running or to cover different variables / plot some other things). Storing them in the plot folder of the recipe will have users looking through multiple folders, adding a lot of confusion for no real gain.
Plots are overwritten without warning: this is intended for the monitoring use, as we would like to replace the older plots with the newer ones.
Actual path and filename of the plots is configurable on the recipe so users can choose where and how store their plots. This is also mean to facilitate collaboration between colleagues and interacting with presentation apps

Documentation: https://esmvaltool--2184.org.readthedocs.build/en/2184/recipes/recipe_monitor.html

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 Any changed dependencies have been added or removed correctly
🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

🧪 Recipe runs successfully
🧪 Recipe is well documented
🧪 Figure(s) and data look as expected from literature
🛠 Provenance information has been added

jvegreg · 2021-05-31T10:42:22Z

One question, @stefsmeets : the html output is expected to fail if some of the recipe products are outside the recipe folder or that is an unexpected behaviour?

stefsmeets · 2021-05-31T11:20:38Z

@jvegasbsc What do you mean by 'fail'? The html output uses the provenance system to get the paths to where the plots / data are located. I understood these would always be within the recipe output directory, so the paths are relative to the root directory. I don't know what happens if they are outside 😅

jvegreg · 2021-05-31T11:24:32Z

@jvegasbsc What do you mean by 'fail'? The html output uses the provenance system to get the paths to where the plots / data are located. I understood these would always be within the recipe output directory, so the paths are relative to the root directory. I don't know what happens if they are outside 😅

Now you know:

Traceback (most recent call last):
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/_main.py", line 433, in run
    fire.Fire(ESMValTool())
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/_main.py", line 410, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/_main.py", line 105, in process_recipe
    recipe.write_html_summary()
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/_recipe.py", line 1446, in write_html_summary
    output.write_html()
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/experimental/recipe_output.py", line 145, in write_html
    html_dump = self.render(template=template)
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/experimental/recipe_output.py", line 161, in render
    rendered = template.render(
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/experimental/templates/recipe_output_page.j2", line 12, in top-level template code
    {% include 'RecipeOutput.j2' %}
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/experimental/templates/RecipeOutput.j2", line 3, in top-level template code
    {% include 'TaskOutput.j2' %}
  File "/home/Earth/jvegas/git/ESMValCore/esmvalcore/experimental/templates/TaskOutput.j2", line 9, in top-level template code
    <img width='80%' src='{{ file.path.relative_to(session.session_dir) }}' alt='{{ file.caption }}'/>
  File "/home/Earth/jvegas/.conda/envs/esmvaltool/lib/python3.9/pathlib.py", line 928, in relative_to
    raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '/home/Earth/jvegas/plots/EC-Earth3-CC/piControl/seaIce/sivol/monclimarctic_sivol_EC-Earth3-CC_SImon_piControl_r1i1p1f1.png' is not in the subpath of '/esarchive/scratch/jvegas/recipe_monitor_20210531_102719' OR one path is relative and the other is absolute.

I will open an issue later to discuss how to manage this, but now I need to go cooking!

stefsmeets · 2021-05-31T12:04:33Z

@jvegasbsc Yeah, best to make an issue for that. This fails because if the output plot is not in a subdirectory of the recipe output. This is to guarantee that the html still points to the right data / png files when the directory is moved.

I wonder if it is possible to keep the plots/data inside the directory, and simply link to it from the outside directory.

jvegreg · 2021-05-31T12:13:33Z

The job I use directly deletes the recipe folder if everything works so that is not possible.

stefsmeets · 2021-05-31T12:22:47Z

Can you add a link the other direction then? 😅

… monitoring Conflicts: setup.py

… monitoring

Co-authored-by: Valeriu Predoi <valeriu.predoi@gmail.com>

sloosvel · 2022-02-18T10:44:58Z

@valeriupredoi I have reworked a bit the documentation and added comments in the eofs script, is it clearer now?

sloosvel · 2022-02-21T10:20:39Z

If this diag cannot make it, can we at least get mapgenerator into the tool?

valeriupredoi · 2022-02-21T13:29:28Z

If this diag cannot make it, can we at least get mapgenerator into the tool?

looking at the changes now, Saskia, thanks much for them! We will get this in - no worries, we have a week before the big release, and even then I suspect there may be wiggles with other bits of the Tool that may delay the release by few days. This is of very good contributing nature so the rules can be bent a bit for it to get in, right @schlunma ? 😁

valeriupredoi

couple minor bits but overall rather happy, many thanks @sloosvel 🍺 @schlunma would you maybe have time to test this plss? Also, we need to remember to include mapgenerator>=1.0.5 in the conda recipe 👍

doc/sphinx/source/recipes/recipe_monitor.rst

esmvaltool/diag_scripts/monitor/compute_eofs.py

Co-authored-by: Valeriu Predoi <valeriu.predoi@gmail.com>

schlunma · 2022-02-23T08:38:29Z

Having a look at this now

schlunma

Really nice addition, thanks @sloosvel! I run the recipe successfully (also added other models from other projects). Some of the plots have overlapping text, but since this is highly customizable I don't think that's an issue now.

Apart from very minor comments on the doc, I have two suggestions/questions:

I think it might be great to mention in the doc that this recipe can plot arbitrary model output. This is a feature that many users asked for. Maybe in the FAQ? Maybe something like "Can ESMValTool plot arbitrary model output?" and a link to this recipe?
I noticed that some plots are in .svg format even though I use output-file-type=png. It looks like this is hardcoded in some parts of the diagnostics. What's the reason for that? I think we should stick to the format the user asks for.

doc/sphinx/source/recipes/recipe_monitor.rst

schlunma · 2022-02-23T08:51:19Z

doc/sphinx/source/recipes/recipe_monitor.rst

+----------
+
+  * plots:
+    a dictionary containing the plots to make, with its own options.


Are you referring to the plot types listed above and the options listed below? Maybe add a link to the section above (List of plot types available in monitor.py) and below (`Plot specific options´) if that's the case.

Really nice addition, thanks @sloosvel! I run the recipe successfully (also added other models from other projects). Some of the plots have overlapping text, but since this is highly customizable I don't think that's an issue now.

Javi's work mostly!

I think it might be great to mention in the doc that this recipe can plot arbitrary model output. This is a feature that many users asked for. Maybe in the [FAQ]
(https://github.com/ESMValGroup/ESMValTool/blob/main/doc/sphinx/source/faq.rst)? Maybe something like "Can ESMValTool plot arbitrary model output?" and a link to this recipe?

Good idea, I will add that

I noticed that some plots are in .svg format even though I use output-file-type=png. It looks like this is hardcoded in some parts of the diagnostics. What's the reason for that? I think we should stick to the format the user asks for.

Can't think of any reason, I guess is something pending to be cleaned. Will update.

I noticed that some plots are in .svg format even though I use output-file-type=png. It looks like this is hardcoded in some parts of the diagnostics. What's the reason for that? I think we should stick to the format the user asks for.

Can't think of any reason, I guess is something pending to be cleaned. Will update.

They are hardoced for size and looks reasons:

svg can be zoomed in and out without losing quality, so it is prefered by default

svg maps are enormous, so it had to be done in png format. Timeseries on the other hand are smaller in svg than png, so they were a better fit for the online app this was originally for.

On the second point, probably rasterization would be a perfect fit.

On the second point, probably rasterization would be a perfect fit.

I think that is what is done when saving the files as png

too little time

I do have very little time right now, sorry.

@zklaus welcoming back Javi, and offering support for your rasterization idea - but I reckon we should not dabble around this just here, I think this whole show will need quite a bit more work to be done in various follow-up PRs, so maybe we can discuss all this there - this is big and functional enough (alas, an excellent contribution!) to be merged now

Understandable, @sloosvel. Could you open an issue about it?

Done in #2561

I took the liberty of assigning the 2.6.0 milestone 😊

doc/sphinx/source/recipes/recipe_monitor.rst

Co-authored-by: Manuel Schlund <32543114+schlunma@users.noreply.github.com>

…into monitoring

schlunma

One small additional comment, after that I would merge that! Thanks @sloosvel and @jvegreg 👍

schlunma · 2022-02-25T13:37:10Z

doc/sphinx/source/faq.rst

@@ -108,3 +108,11 @@ a symbolic link to it so it gets picked up at every re-run iteration:
 .. way to have this mode run. The information provided will help you obtain any data
 .. that is missing and/or create fixes for the datasets and variables that failed the
 .. CMOR checks and could not be fixed on the fly.
+
+
+Can the ESMValTool plot arbitrary model output


Suggested change

Can the ESMValTool plot arbitrary model output

Can the ESMValTool plot arbitrary model output?

schlunma · 2022-02-25T13:37:20Z

doc/sphinx/source/faq.rst

+
+
+Can the ESMValTool plot arbitrary model output
+==============================================


Suggested change

==============================================

===============================================

zklaus · 2022-02-25T13:41:36Z

Great addition!

One small comment, not worth any hold-up: I think we should consistently use "ESMValTool" as a proper name, that means no articles in front of it. In other words, it should never be "the ESMValTool", but only "ESMValTool".

valeriupredoi · 2022-02-25T13:54:53Z

Great addition!

One small comment, not worth any hold-up: I think we should consistently use "ESMValTool" as a proper name, that means no articles in front of it. In other words, it should never be "the ESMValTool", but only "ESMValTool".

very good point! I have noticed (and been mildly bothered) by this very many times with a lot of the writeups about the ESMValTool 😁

sloosvel · 2022-02-25T14:39:18Z

Thanks!!!

Javier Vegas-Regidor added 2 commits May 28, 2021 11:26

First try

ea7bc94

Generalize a bit

cb0c2ea

jvegreg added diagnostic is-enes labels May 29, 2021

jvegreg requested review from zklaus, bouweandela and schlunma May 29, 2021 09:41

Javier Vegas-Regidor added 5 commits May 29, 2021 11:45

Add mapgenerator to requirements

41e9a46

Fix tests

607e0a0

Make it work with main

0b56ec1

Fix flake8

77e5139

Add provenance

908388c

jvegreg mentioned this pull request May 31, 2021

HTML report does not work if there are outputs outside the recipe folder ESMValGroup/ESMValCore#1154

Closed

Javier Vegas-Regidor added 11 commits June 21, 2021 10:36

Update mapgenerator version

1f32c25

Merge branch 'main' of https://github.com/ESMValGroup/ESMValTool into…

1eb24a4

… monitoring Conflicts: setup.py

Update recipe to use CMIP6 data

b0fbee1

Merge branch 'main' of https://github.com/ESMValGroup/ESMValTool into…

c76b4a6

… monitoring

Fix doc and add more examples

8326c1c

Some improvements

f7ab6a5

Apply yapf

e0ebbbd

Fix codacy issues

1575df9

Fix some graphical glitches

d22334a

Merge branch 'main' of https://github.com/ESMValGroup/ESMValTool into…

f1f20ee

… monitoring

Add doc for recipe

459a9e3

schlunma added in scientific review in technical review and removed looking for scientific reviewer labels Feb 15, 2022

sloosvel and others added 5 commits February 16, 2022 10:29

Apply suggestions from code review

85381ce

Co-authored-by: Valeriu Predoi <valeriu.predoi@gmail.com>

Comment eofs code

35fe957

Improve doc

b4cdc47

Merge remote-tracking branch 'origin/main' into monitoring

1867d83

Fix style issues

337ddbe

valeriupredoi approved these changes Feb 21, 2022

View reviewed changes

doc/sphinx/source/recipes/recipe_monitor.rst Outdated Show resolved Hide resolved

esmvaltool/diag_scripts/monitor/compute_eofs.py Show resolved Hide resolved

Update doc/sphinx/source/recipes/recipe_monitor.rst

2d6874a

Co-authored-by: Valeriu Predoi <valeriu.predoi@gmail.com>

schlunma reviewed Feb 23, 2022

View reviewed changes

sloosvel and others added 6 commits February 23, 2022 16:31

Apply suggestions from code review

673059c

Co-authored-by: Manuel Schlund <32543114+schlunma@users.noreply.github.com>

Add entry in FAQ

8098190

Reference list of plots

0f86a5a

Merge branch 'monitoring' of https://github.com/ESMValGroup/ESMValTool …

5079251

…into monitoring

Update reference

8aa1051

Update reference

3fc208a

sloosvel mentioned this pull request Feb 25, 2022

Format of output plots in recipe_monitor.yml is sometimes hardcoded instead of configurable #2561

Closed

schlunma reviewed Feb 25, 2022

View reviewed changes

Improve doc

0a1aea0

schlunma approved these changes Feb 25, 2022

View reviewed changes

schlunma merged commit b68efbf into main Feb 25, 2022

schlunma deleted the monitoring branch February 25, 2022 14:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recipe to plot generic output from the preprocessor #2184

Recipe to plot generic output from the preprocessor #2184

jvegreg commented May 29, 2021 •

edited by schlunma

Loading

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021 •

edited

Loading

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021 •

edited

Loading

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021

sloosvel commented Feb 18, 2022

sloosvel commented Feb 21, 2022

valeriupredoi commented Feb 21, 2022

valeriupredoi left a comment

schlunma commented Feb 23, 2022

schlunma left a comment

schlunma Feb 23, 2022

sloosvel Feb 25, 2022

jvegreg Feb 25, 2022

zklaus Feb 25, 2022

jvegreg Feb 25, 2022

sloosvel Feb 25, 2022

valeriupredoi Feb 25, 2022 •

edited

Loading

zklaus Feb 25, 2022

sloosvel Feb 25, 2022

zklaus Feb 25, 2022

schlunma left a comment

schlunma Feb 25, 2022

schlunma Feb 25, 2022

zklaus commented Feb 25, 2022

valeriupredoi commented Feb 25, 2022

sloosvel commented Feb 25, 2022

	Can the ESMValTool plot arbitrary model output
	Can the ESMValTool plot arbitrary model output?



		Can the ESMValTool plot arbitrary model output
		==============================================

Recipe to plot generic output from the preprocessor #2184

Recipe to plot generic output from the preprocessor #2184

Conversation

jvegreg commented May 29, 2021 • edited by schlunma Loading

Checklist

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021 • edited Loading

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021 • edited Loading

jvegreg commented May 31, 2021

stefsmeets commented May 31, 2021

sloosvel commented Feb 18, 2022

sloosvel commented Feb 21, 2022

valeriupredoi commented Feb 21, 2022

valeriupredoi left a comment

Choose a reason for hiding this comment

schlunma commented Feb 23, 2022

schlunma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valeriupredoi Feb 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schlunma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zklaus commented Feb 25, 2022

valeriupredoi commented Feb 25, 2022

sloosvel commented Feb 25, 2022

jvegreg commented May 29, 2021 •

edited by schlunma

Loading

stefsmeets commented May 31, 2021 •

edited

Loading

stefsmeets commented May 31, 2021 •

edited

Loading

valeriupredoi Feb 25, 2022 •

edited

Loading