change in ESMValTool breaks eWaterCycle forcing

[RolfHut]

I think a recent change in ESMValTool breaks the eWaterCycle forcing module. Anytime you import ewaterCycle.forcing you get the following error:

ModuleNotFoundError                       Traceback (most recent call last)
File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:9
      8 try:
----> 9     import ESMF as esmpy  # noqa: N811
     10 except ImportError:

ModuleNotFoundError: No module named 'ESMF'

ESMF is not a module that can be (easily?) installed using pip.

This means that currently on researchcloud (and other places) we can not work with eWaterCycle.

[Peter9192]

Yes, I also noticed that in eWaterCycle/infra#137.

[BSchilperoort]

The source of this issue is ESMF changing their package's module name (in a minor release...).

See here:
"The Python module name for ESMPy was changed in this release from “ESMF” to “esmpy”"

By instead installing esmpy<8.4.0 with conda/mamba, this issue should be (temporarily) fixed.

[RolfHut]

esmpy can't be installed

conda install 'esmpy<8.4.0'
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - esmpy[version='<8.4.0']

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

However: if we switch to the eWaterCycle conda environment, it does work. However, we are not sure how to link this environment to notebooks?

[BSchilperoort]

I forgot to specify that the channel is conda-forge.

conda install --channel=conda-forge "esmpy<8.4"

should work. I just tested it (with micromamba).

Except if you are on Python 3.11, then you will receive the following error:

    └─ pin-1   is uninstallable because it requires
       └─ python 3.11.* , which conflicts with any installable versions previous

If you want to create a new environment that is compatible with the current release of eWaterCycle, you should be able to use the following environment.yml. The python version is pinned to 3.10 here.

---
name: ewatercycle
channels:
  - conda-forge
dependencies:
  - python ==3.10
  - esmvaltool-python >=2.3.0
  - subversion
  - esmpy !=8.1.0,<8.4

[RolfHut]

I'm trying to find where this yml file lives in the jupyterlab environment and to change it such that all user benefit from it. Any suggestions?

[Peter9192]

You mean this one? /opt/conda/environment.yml?
See https://github.com/eWaterCycle/infra/blob/main/roles/ewatercycle/defaults/main.yml and also https://github.com/eWaterCycle/infra/blob/main/roles/ewatercycle/files/environment.yml.

[RolfHut]

do I need to restart the machine? (if so, how?) or do I need to do something else to make the changes work? Since I still do not see the environment become available in jupyterlab. I did try this : https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook/56409235#56409235 but it doesn't add the environment to my options in jupyterlab

[sverhoeven]

https://github.com/eWaterCycle/infra/blob/main/roles/ewatercycle/files/environment.yml