Enhancement of the hyint recipe to include etccdi indices

[earnone]

This version of hyint now includes the recipe_hyint_extreme_events with the hyint diagnostics in pipe with the extreme_events diagnostics. See #1063. This capability was already in the previous version of hyint but not called, so only a few refinements were applied.

---

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)

• Give this pull request a descriptive title that can be used as a one line summary in a changelog

• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.

• Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.

• Please use yamllint to check that your YAML files do not contain mistakes
  New recipe/diagnostic

• Add documentation for the recipe to the doc/sphinx/source/recipes folder and add a new entry to index.rst

• Add provenance information

---

Closes #1063

[bouweandela]

Hi @earnone, is this something you are still working on, or is it ready for merging from your point of view?

[earnone]

@bouweandela it needs a little update to the provenance piped from extreme_events.

[bouweandela]

@earnone Can you finalize this pull request? Then we can test it when you're done.

[bouweandela]

@earnone We recently started using the styler package to style all our R code. Of course this introduced some conflicts with the changes implemented here, which I just tried to fix. Can you please check if your changes are still as expected (and use the styler package from now on to style your code?).

[mattiarighi]

@earnone is this still relevant?
If yes, please finalize, otherwise close it and remove the branch.

[earnone]

Sorry I kept this on stand by because of the cdo issue but this is now solved. Yes, will finalize quickly as it was actually already working. Thanks @bouweandela <https://github.com/bouweandela> for the styler, yes will check everything is working.

…

On Thu, 19 Dec 2019 at 10:58, Mattia Righi ***@***.***> wrote: @earnone <https://github.com/earnone> is this still relevant? If yes, please finalize, otherwise close it and remove the branch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1133?email_source=notifications&email_token=AG4WXMN3AOYNTT5KAZBWT2LQZNATTA5CNFSM4HTWUUH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJCMQA#issuecomment-567420480>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4WXMICWGNMSI4SPWJQDVTQZNATTANCNFSM4HTWUUHQ> .

[earnone]

I am closing this and opening a new pull request from scratch starting from master.

[mattiarighi]

Please also delete the branch, if not used.

[earnone]

@bouweandela @mattiarighi I have reopened this to finalize the extreme_events option of hyint. No need for a new pull request. Solving conflicts with latest version and working on provenance in cascade from extreme_events (#1063)

[earnone]

All tests were successful. @bouweandela could you test it and review this? Thanks.

[bouweandela]

Looks good! Could you please also add documentation for the new recipe and give the pull request a descriptive title that I can use in the changelog when making a new release of ESMValTool?

[earnone]

@bouweandela The current recipe_hyint.rst in master already mentions the possibility of including additional indices from extreme_events. In fact, the recipe_hyint_extreme_events.yml is just as recipe_hyint.yml but with the further call to the extreme_events diagnostic and inclusion of its results in hyint.
I would like to avoid to cause confusion, would it be better to point to a joint documentation? Or do we want to duplicate everything and add the further specs for the extreme_events part?

[bouweandela]

In that case you could maybe update recipe_hyint.rst so it includes this recipe? E.g. mention it here: https://github.com/ESMValGroup/ESMValTool/blob/master/doc/sphinx/source/recipes/recipe_hyint.rst#available-recipes-and-diagnostics and add some explanation on what the difference between the two recipes is?

[earnone]

Yes, I'd prefer to keep all in one place. How does it work with the file recipe_hyint_extreme_events.rst? We simply don't create it?

[earnone]

The recipe_hyint.rst file was updated to cover also the options for recipe_hyint_extreme_events.yml (without creating a separate recipe_hyint_extreme_events.rst). This should complete the PR.

[bouweandela]

Looks good to me now, thanks! @mattiarighi Could you please test and merge when you're happy with this?

[mattiarighi]

extreme_events.R stops because of a missing library:

Error in library(ncdf4.helpers) :
  there is no package called ‘ncdf4.helpers’
Execution halted

[earnone]

extreme_events.R stops because of a missing library:
Error in library(ncdf4.helpers) :`

Yes, this is the issue affecting extreme_events.R (lack of ncdf4.helpers library on CRAN and need to install it from archive) and the reason for keeping this hyint capability in stand-by. I understand we decided to proceed anyway while waiting for the library to be upgraded or extreme_events rewritten without the missing library?

[mattiarighi]

Sorry, I missed the discussion.

Should I test without extreme_events.R for now and leave this to another PR?

[earnone]

Sorry, I missed the discussion.

Should I test without extreme_events.R for now and leave this to another PR?

Actually this improvement of hyint is solely related to ingesting the results from extreme_events, so there is nothing to test without that part (everything else is untouched). As you prefer, we can also keep this aside and not include it in master for now. @bouweandela?

[bouweandela]

@mattiarighi The problem is reported here #1443. I would suggest going ahead with testing and merging this pull request, because the figures are part of the paper and the problem is with an existing recipe, not so much with what is added in this pull request.

We can see how to proceed with the dependency problems later, either by re-implementing the indices or by helping the authors of the ncdf4.helpers package to get their package back on CRAN.

[mattiarighi]

Dumb question: where can I get the `ncdf4.helpers' package to test this PR?

[earnone]

Dumb question: where can I get the `ncdf4.helpers' package to test this PR?

Here

And here is a suggestion on how to install from url:

# Download package tarball from CRAN archive
url <- "https://cran.r-project.org/src/contrib/Archive/ncdf4.helpers/ncdf4.helpers_0.3-3.tar.gz"
pkgFile <- "ncdf4.helpers_0.3-3.tar.gz"
download.file(url = url, destfile = pkgFile)
# Install package
install.packages(pkgs=pkgFile, type="source", repos=NULL)
# Delete package tarball
unlink(pkgFile)

[mattiarighi]

Thanks, I have tested both recipes successfully.

Maybe you can add the above snippet to the recipe documentation until we solve #1443.

[earnone]

@mattiarighi Could you have a look? Do we have a standard way to include known issues or warnings in the doc? I can replicate that in recipe_extreme_events too.

[mattiarighi]

Thanks, it looks good to me.

I don't think we have a standard way of documenting this. Another option would be to add this to the recipe description section, it's more likely that users would look at that.

@bouweandela any opinion?

[bouweandela]

I think it's fine like this: we do not have a standard for documenting workarounds, because we try to just fix bugs instead. However, in this case that is taking some time, unfortunately.