error with cfgrib + eccodes

[ocarino]

Hi, I tried to install cfgrib and eccodes on two different conda installations. Installation is always successful. Right now I do have:

% conda list | egrep "(eccodes|cfgrib)"
cfgrib                    0.9.9.0            pyhd8ed1ab_0    conda-forge
eccodes                   2.21.0               ha0e6eb6_0    conda-forge
python-eccodes            2021.03.0        py38hb5d20a5_1    conda-forge

Importing xarray works, but whenever I do an "import cfgrib" or an "ds = xarray.open_dataset('gribfile.grb', engine='cfgrib')", I get:

self.ds = cfgrib.open_file(filename, **backend_kwargs) AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import)

(full error output attached). This happens on both conda installations (one is miniforge3-4.9.2.3, the other one is anaconda3 conda version 4.10).

Any idea what can cause this issue? FYI: I just downgraded cfgrib to 0.9.8.5 and now it works.

Cheers, Olaf

[keewis]

We would need more information about your environments. Could you post the output of xr.show_versions()? Also, can you reproduce this with

xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")

[aurghs]

I can try to reproduce the error and fix it, but I need to know at least the xarray version.

[raybellwaves]

Not sure if conda-forge/python-eccodes-feedstock#71 helps
Came across this via ecmwf/cfgrib#226

[ocarino]

conda-error-output.txt
conda-xr-showversions.txt

Hi keewis and aurghs, as the working version with cfgrib 0.9.8.5 is already in production, it took some time to reproduce the previous environment from a backup. So here we are. Versions are:

% conda list | egrep "(cfgrib|xarray|eccodes)"
cf_xarray 0.3.1 pyhd3deb0d_0 conda-forge
cfgrib 0.9.9.0 pyhd8ed1ab_1 conda-forge
eccodes 2.21.0 ha0e6eb6_0 conda-forge
python-eccodes 2021.03.0 py38hb5d20a5_1 conda-forge
xarray 0.16.2 pyhd8ed1ab_0 conda-forge

With "xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")" the result is the same, error message is:

AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import)

Full error output is attached. And I attached the output of "xr.show_versions()". I wonder why it shows "cfgrib: None"...

Cheers, Olaf

[ocarino]

And I have to correct myself: Though I still have an output of xr_open_dataset from cfgrib version 0.9.8.5 that shows that it worked (otherwise I would doubt on my senses), it does not work right now. xr.show_versions shows "cfgrib: None", too.

Also "import cfgrib" shows the same error. This sounds much like ecmwf/cfgrib#226 , but obviously I already use eccodes 2.21.0 (which is the proposed fix there...)

[keewis]

I still can't reproduce this. For reference, here's my environment:

output of xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) 
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 4.19.0-16-amd64
machine: x86_64
processor: 
byteorder: little
libhdf5: None
libnetcdf: None

xarray: 0.16.2
pandas: 1.2.4
numpy: 1.20.2
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: 0.9.9.0
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 49.6.0.post20210108
pip: 21.0.1
conda: 4.10.1
pytest: None
IPython: 7.22.0
sphinx: None

And I attached the output of "xr.show_versions()"

I can't seem to find that. Am I missing something?

xr.show_versions shows "cfgrib: None", too.

I guess that's because the import fails.

[ocarino]

I added the missing attachments to my previous comment, seems that my browser did not cope with the upload mechanism.

xr.show_versions shows "cfgrib: None", too.

I guess that's because the import fails.

Well, just to narrow down the error: I guess a working "import cfgrib" would be the precondition for it to work with xarray... (?)

[keewis]

strange. This seems like a environment issue. Could you try creating a environment with conda env create -f env.yml?

contents of env.yml

name: test
channels:
  - conda-forge
dependencies:
  - python=3.8.6
  - xarray=0.16.2
  - pandas=1.2.1
  - numpy=1.19.2
  - scipy=1.6.0
  - netCDF4=1.5.5.1
  - h5py=2.10.0
  - zarr=2.6.1
  - cftime=1.3.1
  - nc-time-axis=1.2.0
  - iris=2.4.0
  - bottleneck=1.3.2
  - dask=2021.01.0
  - distributed=2021.01.0
  - matplotlib=3.3.3
  - cartopy=0.18.0
  - seaborn=0.11.1
  - setuptools=49.6.0
  - pip=20.3.1
  - conda=4.9.2
  - pytest=6.2.1
  - ipython=7.19.0
  - sphinx=3.4.3
  - libnetcdf=4.7.4
  - cfgrib=0.9.9.0
  - h5netcdf=0.8.1

With that, python -c 'import xarray as xr; xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")' passes for me.

[ocarino]

Strange. I deployed a completely new conda (using the miniforge installer Miniforge3-4.9.2-3-Linux-x86_64.sh). With that I created an environment "test" with your yml file. When I run the xr.tutorial.open_dataset (exact copy-paste from your message), I get the same error as usual:

AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import

What the heck can be the problem that I face? I am bemused...

[keewis]

did you try mamba or conda from miniconda?

Edit: also, did you make sure to activate the new environment (check $PYTHONPATH and which python) and that you are not in a directory containing the code of xarray or cfgrib?

[ocarino]

Hi, keewis wrote on 15.04.21 17:51:

did you try |mamba| or |conda| from |miniconda|?

right now I only did miniforge (though our elder python environments are anaconda or conda). And: Yes, the env was activated and no other xarray or cfgrib stuff around here. :-) When I create a complete new miniforge3-4.9.2-3 on our system and only do a conda install eccodes conda install cfgrib the thing works. So there must be some relicts from earlier package installs or some other versions that cause the trouble. I got some hints from a different site that it might even be dependent on the order in which things are installed, not sure about that. Still I wonder that the error message seems to imply that there might be a circular import... I'll do some further testing and will come back when I have something that is more concrete so that you can look at it. Thanks, Olaf

…

-- Dipl. Inform. Olaf Gellert email ***@***.*** Deutsches Klimarechenzentrum GmbH phone +49 (0)40 460094 214 Bundesstrasse 45a fax +49 (0)40 460094 270 D-20146 Hamburg, Germany www http://www.dkrz.de Sitz der Gesellschaft: Hamburg Geschäftsführer: Prof. Dr. Thomas Ludwig Registergericht: Amtsgericht Hamburg, HRB 39784

[ocarino]

Ok, PEBKAC (problem exists between keyboard and chair). Your hints already pointed in the correct direction, I had a testscript in my homedir which was named cfgrib.py, a very unfortunate naming, I'd say... Issue can be closed, everything is working fine. And many thanks for your efforts. Olaf