Align ESMValTool to ESMValCore=2.2.0 (adopt iris3, fix environment for new Core release)

[stefsmeets]

This PR tracks the progress to use iris3 in esmvalcore/esmvaltool. See this issue: #1862
Using this branch of esmvalcore: https://github.com/ESMValGroup/ESMValCore/tree/iris3 (PR: ESMValGroup/ESMValCore#819)

To do list

We should keep this PR as is until ESMValGroup/ESMValCore#819 is merged and released(?). Before merging into master, we need to look at:

• Merge master, and address conflicts with code using the updated syntax.
• setup.py: Uncomment 'scitools-iris=3.0.0'

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• Give this pull request a descriptive title that can be used as a one line summary in a changelog
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes

Closes #1862 and #2006

[bouweandela]

I tested:

• examples/recipe_python.yml
• examples/recipe_preprocessor_test.yml

with this branch and compared the runtime and results with the master branch.

The runtime is similar and the resulting netcdf files are too, though there was some minor difference in the time axis of the multimodel statistics of diagnostic_1 of the preprocessor test recipe (different calendar chosen from input cubes), but I suspect that is related to the multi model statistics preprocessor and not an iris issue.

[valeriupredoi]

something to be aware of #2006

[valeriupredoi]

failing tests are dubious, here SciTools/iris#3967

[valeriupredoi]

failing tests are dubious, here SciTools/iris#3967

@schlunma you mind fixing those tests in line with the change of iris.coords.Coord as @bjlittle tells us about in SciTools/iris#3967 🍺

[schlunma]

Will do!

[schlunma]

Haha, so apparently it was me who coded that 😂 No idea why I did that though...

[valeriupredoi]

Haha, so apparently it was me who coded that joy No idea why I did that though...

shh, nobody know 🤣 We need to grep on the diags dir see if others shared your penchant for Coord() 😁

[schlunma]

Just checked it, it's only me 😂

But thank god we had unit tests for this...I think we need to thoroughly test all diagnostics before releasing this, I'm pretty sure there are other issues waiting for us not tested by unit tests.

[schlunma]

Just fixed the tests 👍

I just noticed that I get the following warning every time I import iris now:

Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import iris
/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm2/lib/python3.7/site-packages/iris/config.py:139: UserWarning: Ignoring config item 'Resources':'test_data_dir' (section:option) as '/home/h05/itwl/projects/git/iris-test-data/test_data' is not a valid directory path.
  warnings.warn(msg.format(section, option, c_path))

Anyone else experiencing this? If yes I will open an issue in the iris repo.

[schlunma]

Would it be possible to get this merged as soon as possible? For a new PR I need the new release branch of ESMValCore (or its master), and the current master of ESMValTool does not work with that because of iris3!

[valeriupredoi]

Would it be possible to get this merged as soon as possible? For a new PR I need the new release branch of ESMValCore (or its master), and the current master of ESMValTool does not work with that because of iris3!

yes, let me pin mpich from #2007 then test and will merge - will do this tomorrow morning 👍

[bouweandela]

The changes in this pull request need ESMValCore v2.2, so this can only be merged once that has been released (next Monday).

@schlunma You can just branch off of this branch for your pull request that needs the development version of the ESMValCore.

[schlunma]

Cool, thanks!! I already branched of this branch, just wanted to make sure that this PR has priority

[valeriupredoi]

I just re-double-checked the situation with mpich that needs to be pinned to avoid library import issues, even though the Core from release _2.2.0 installs a newer version of mpich:

(release22x) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda list mpich
# packages in environment at /home/valeriu/miniconda3/envs/release22x:
#
# Name                    Version                   Build  Channel
mpich                     3.4.1                h846660c_2    conda-forge

the updated env for Tool with this branch updates the mpich to the offending package:`

(release22x) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ conda list mpich
# packages in environment at /home/valeriu/miniconda3/envs/release22x:
#
# Name                    Version                   Build  Channel
mpich                     3.4.1                external_2    conda-forge

that's causing us issues. So the pin needs to stay in for now. I will add a note to the env and meta file so we remove it as soon as we can. Gotta love conda

[valeriupredoi]

OK I've also pinned to the new esmvalcore that will be released today by the magnificent @jvegasbsc 💃

[bouweandela]

It's important that you keep the pin on the upper version of esmvalcore, or the esmvaltool package may end up being broken in the two weeks between the esmvalcore release and esmvaltool release.

[valeriupredoi]

It's important that you keep the pin on the upper version of esmvalcore, or the esmvaltool package may end up being broken in the two weeks between the esmvalcore release and esmvaltool release.

very good point @bouweandela cheers 🍺

[bouweandela]

Merged #1994 into this branch to see if it makes the tests pass.

[bouweandela]

The conda build is failing, but it seems unrelated to this pull request, see #2016.

Because we really need this to move forward with testing of ESMValTool for the upcoming release and integrating final features, I will merge this.

[valeriupredoi]

yes yes yes 🍺 And the install works a-ok with esmvalcore=2.2.0 👍