ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support #47

yongjias · 2019-03-13T19:12:45Z

Hello, I installed successfully esmf_7_1_0r with ESMPy using option ESMF_COMM=mpiuni. However, I got the error "Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support" when I use xesmf "regridder = xe.Regridder(grid_in, grid_out, 'bilinear')". I wonder if the ESMF has to been compiled by MPI option?

JiaweiZhuang · 2019-03-13T19:15:50Z

How did you get ESMPy? Simply conda install -c conda-forge esmpy should work. Did you compile it from source?

yongjias · 2019-03-13T19:23:28Z

I compiled ESMPy from ESMF 7.1 source code ( addon directory ). thanks,

JiaweiZhuang · 2019-03-13T19:26:53Z

Could you see if conda install works? So many things can happen when compiling from source.

yongjias · 2019-03-13T19:27:40Z

I will try conda install and report here later. thanks

yongjias · 2019-03-14T19:31:37Z

Installation by conda solved the problem. Thanks.

JiaweiZhuang · 2019-03-14T19:43:14Z

Great!

ks905383 · 2019-12-05T05:43:59Z

Hi @JiaweiZhuang , I ran into this same issue / error message while trying to get a jupyter notebook to work with binder. I'm explicitly including mpi4py and esmf in addition to xesmf in the environment file (which should be conducting the conda install as you mention in this thread, if I understand correctly, too), to no avail. Do you know what might be causing that?

The Binder is here (based on the repo https://github.com/ks905383/impact-variability):
https://mybinder.org/v2/gh/ks905383/impact-variability/234b4747e7f402b843e70a68499774aa761f9f00

The relevant cell is in the master_run file notebook, in the code section "Preprocessing Historical Data";
# Regrid to MPI-GE grid
regridder = xe.Regridder(data_era_hist,data_mpi_hist,'bilinear')

Thank you!

JiaweiZhuang · 2019-12-05T14:58:35Z

@ks905383 That's probably because you are mixing conda-forge and defaults channels:

channels:
    - conda-forge
    - defaults

See #54 (comment) and conda-forge/esmpy-feedstock#28

Also, does it work if you remove mpi4py?

ks905383 · 2019-12-06T02:16:19Z

Hi Jiawei,

Thanks for the quick reply. Unfortunately, this doesn't seem to be the issue (I had conda-forge listed first; I tried listing it as the only channel, without success. I tried a few other things unsuccessfully as well - removing mpi4py, putting esmf first as was suggested in one of the threads, etc.).

bradyrx · 2019-12-10T18:07:58Z

FYI the only thing that worked for me was to force esmpy<=7.1.0 in my conda environment file as in conda-forge/esmpy-feedstock#28

lastproxy · 2019-12-30T16:44:18Z

FYI the only thing that worked for me was to force esmpy<=7.1.0 in my conda environment file as in conda-forge/esmpy-feedstock#28

downgrading esmpy to 7.1.0 solved my same issue..

matteodefelice · 2020-01-23T08:03:34Z

Same problem to me, just installed xesmf from conda-forge. Unfortunately I cannot downgrade esmpy without breaking my environment (I get UnsatisfiableError when trying installing esmpy 7.1).

JiaweiZhuang · 2020-01-23T15:08:04Z

(I get UnsatisfiableError when trying installing esmpy 7.1).

Would you be able to create a new conda environment?

brianpm · 2020-01-28T23:18:39Z

Just want to report that I ran into the same problem. I was unable to get xesmf (0.2.1) to work with esmpy=8.0.0. Creating a new conda environment with esmpy=7.1.0 and xesmf=0.2.1 seems to work.

stefan-hofer · 2020-02-05T14:02:32Z

I also want to say that I ran into the same problem, and it was solved by going back to esmpy=7.1.0 and xesmf=0.2.1

It seems that MPI isn't installed with esmpy=8.0.0 and esmf=8.0.0, which might explain the error message ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support

## Package Plan ##

  environment location: /uio/kant/geo-metos-u1/shofer/.conda/envs/shofer

  added / updated specs: 
    - esmpy=7.1.0
    - xesmf=0.2.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mpich-3.3.2                |       hc856adb_0         6.4 MB  conda-forge
    netcdf-fortran-4.4.5       |    h0789656_1004         1.2 MB  conda-forge
    mpi-1.0                    |            mpich           4 KB  conda-forge
    esmf-7.1.0                 |    hcda5390_1008        23.7 MB  conda-forge
    esmpy-7.1.0                |   py38h5ca1d4c_3         161 KB  conda-forge
    libnetcdf-4.6.2            |    h303dfb8_1003         1.3 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        32.7 MB

The following NEW packages will be INSTALLED:

    mpi:            1.0-mpich                    conda-forge
    mpich:          3.3.2-hc856adb_0             conda-forge

The following packages will be REMOVED:

    netcdf4:        1.5.3-nompi_py38hd35fb8e_102 conda-forge

The following packages will be DOWNGRADED:

    esmf:           8.0.0-nompi_he6d0a24_4       conda-forge --> 7.1.0-hcda5390_1008  conda-forge
    esmpy:          8.0.0-nompi_py38hf0e99fa_1   conda-forge --> 7.1.0-py38h5ca1d4c_3 conda-forge
    libnetcdf:      4.7.3-nompi_h9f9fd6a_101     conda-forge --> 4.6.2-h303dfb8_1003  conda-forge
    netcdf-fortran: 4.5.2-nompi_h09cde99_103     conda-forge --> 4.4.5-h0789656_1004  conda-forge

mickaellalande · 2020-03-02T10:04:28Z

I went through the same issue but I actually was not satisfied with any of the previous solutions... because having esmpy<=7.1.0 was making me stuck for installing other packages, and the solution from (#55, #68, ECMWFCode4Earth/ml_drought#71) to set LD_LIBRARY_PATH='' was not solving the problem in my case. After many tries, I finally found a simple solution thanks to this comment:

It seems that MPI isn't installed with esmpy=8.0.0 and esmf=8.0.0

And fortunately, I succeeded to get MPI to install with esmpy last version, and the solution is simple:

conda create -n test
conda activate test
conda install esmpy
conda install xesmf dask
Install whatever other packages you need

If you install them together it doesn't work, if you install xesmf before it doesn't work. There is only by starting the installation of esmpy first that will install the MPI packages (check out that you have well these lines before to install):

mpi                       1.0                       mpich    conda-forge
mpi4py                    3.0.3            py38hcf07815_0    conda-forge
mpich                     3.3.2                hc856adb_0    conda-forge

That way I have all the test that passed (https://xesmf.readthedocs.io/en/latest/installation.html#testing-your-installation):

$ pip install pytest
$ pytest -v --pyargs xesmf  # should all pass

####################################################################
My Miniconda installation:

This solution will maybe not working for everyone and it is probably linked to this version and my environment. So here is my Miniconda installation method:

Install Miniconda (https://docs.conda.io/en/latest/miniconda.html):

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc

Add conda-forge (https://conda-forge.org/docs/user/introduction.html):

conda config --add channels conda-forge
conda config --set channel_priority strict
conda update -n base -c defaults conda

The key is after to install esmpy as the first package in a new environment; because I don't know which other packages block it to install the MPI libraries, but on existing environments when I try to install esmpy by itself it does or doesn't install the MPI libraries...

Here is the conda list:

# packages in environment at /data/mlalande/miniconda3/envs/test4:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
attrs                     19.3.0                   pypi_0    pypi
bokeh                     1.4.0                    py38_0    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28               py38_0    conda-forge
click                     7.0                        py_0    conda-forge
cloudpickle               1.3.0                      py_0    conda-forge
curl                      7.68.0               hf8cf82a_0    conda-forge
cytoolz                   0.10.1           py38h516909a_0    conda-forge
dask                      2.11.0                     py_0    conda-forge
dask-core                 2.11.0                     py_0    conda-forge
distributed               2.11.0                   py38_0    conda-forge
esmf                      8.0.0           mpi_mpich_h06e08bd_105    conda-forge
esmpy                     8.0.0           mpi_mpich_py38ha9b28fa_101    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
fsspec                    0.6.2                      py_0    conda-forge
hdf4                      4.2.13            hf30be14_1003    conda-forge
hdf5                      1.10.5          mpi_mpich_ha7d0aea_1004    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
jinja2                    2.11.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
krb5                      1.16.4               h2fd8d38_0    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_8    conda-forge
libblas                   3.8.0               15_openblas    conda-forge
libcblas                  3.8.0               15_openblas    conda-forge
libcurl                   7.68.0               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
liblapack                 3.8.0               15_openblas    conda-forge
libnetcdf                 4.7.3           mpi_mpich_h755db7c_1    conda-forge
libopenblas               0.3.8                h5ec1e0e_0    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libtiff                   4.1.0                hc3755c2_3    conda-forge
llvm-openmp               9.0.1                hc9558a2_2    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py38h516909a_0    conda-forge
more-itertools            8.2.0                    pypi_0    pypi
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.0.3            py38hcf07815_0    conda-forge
mpich                     3.3.2                hc856adb_0    conda-forge
msgpack-python            1.0.0            py38hc9558a2_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
netcdf-fortran            4.5.2           mpi_mpich_hd560429_3    conda-forge
numpy                     1.18.1           py38h95a1406_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
packaging                 20.1                       py_0    conda-forge
pandas                    1.0.1            py38hb3f55d8_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pillow                    7.0.0            py38hefe7db6_0    conda-forge
pip                       20.0.2                     py_2    conda-forge
pluggy                    0.13.1                   pypi_0    pypi
psutil                    5.7.0            py38h516909a_0    conda-forge
py                        1.8.1                    pypi_0    pypi
pyparsing                 2.4.6                      py_0    conda-forge
pytest                    5.3.5                    pypi_0    pypi
python                    3.8.2           h9d8adfe_1_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.3              py38h516909a_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
scipy                     1.4.1            py38h921218d_0    conda-forge
setuptools                45.2.0                   py38_0    conda-forge
six                       1.14.0                   py38_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tblib                     1.6.0                      py_0    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py38h516909a_4    conda-forge
wcwidth                   0.1.8                    pypi_0    pypi
wheel                     0.34.2                     py_1    conda-forge
xarray                    0.15.0                     py_0    conda-forge
xesmf                     0.2.2                      py_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.2.2                h516909a_1    conda-forge
zict                      2.0.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h3b9ef0a_1    conda-forge

spec-file.txt

stefan-hofer · 2020-06-15T15:03:42Z

I just wanted to confirm that @mickaellalande 's solution worked for me. But isn't there a way to fix this issue a bit more robustly? For me I'm running into the exact same problem, and there is no difference whether I'm trying it on an HPC cluster or my personal Linux PC.

bradyrx · 2020-07-28T20:18:53Z

Does conda install from an environment.yml file in the order in which it is listed? If that's the case, maybe one could put esmpy, xesmf, dask as the first three arguments. The old method of keeping esmpy<=7.1.0 isn't working for me anymore but @mickaellalande's works. I just like to avoid manually doing those first few steps when providing a yaml file for developers or users. So ideally there's a way to force this in the environment file.

mickaellalande · 2020-07-29T08:25:59Z

Actually it looks like the order is not important anymore, I've just made a new test thanks to the suggestion of @bradyrx with different .yml files:

Install esmpy first (:+1:):

name: xesmf_v0
channels:
  - conda-forge
  - defaults
dependencies:
  - esmpy # 8.0.1
  - xesmf # 0.3.0
  - dask # 2.21.0

Recommended installation from xESMF webpage (:-1:):

name: xesmf_v1
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.7
  - xesmf
  - dask
  - netCDF4 # not needed because already installed with dask I guess

Other installation without specifying Python version (:-1:):

name: xesmf_v2
channels:
  - conda-forge
  - defaults
dependencies:
  - xesmf
  - dask
  - netCDF4 # not needed because already installed with dask I guess

Install esmpy after xesmf (:+1:):

name: xesmf_v3
channels:
  - conda-forge
  - defaults
dependencies:
  - xesmf
  - esmpy
  - dask

Testing your installation

Works: xesmf_v0, xesmf_v3
Doesn't work: xesmf_v1, xesmf_v2

I also tried back an installation directly with: conda create -n xesmf_v4 xesmf dask esmpy and it works too. So it looks like it is needed to add explicitly esmpy but it doesn't matter anymore about the order, what makes it easier to share.

No change in mpi versions from my last post, and I guess it's still the same problem for xesmf_v2 and xesmf_v3 where esmpy is not specified there is no installation of these following packages:

# packages in environment at /home/lalandmi/miniconda3/envs/xesmf_v0:
#
# Name                    Version                   Build  Channel
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.0.3            py38h4a80816_1    conda-forge
mpich                     3.3.2                hc856adb_0    conda-forge

The only change I see is esmpy that passed from version 8.0.0 to 8.0.1 from my last post (maybe fixing this order issue?).

Here are my environments details with different files formats (.yml with and without the flag --from-history + .txt):
xesmf_envs.zip.

Maybe a solution would be to add the mpi packages as an explicit dependence of xesmf so that it will automatically be installed @JiaweiZhuang? I have no idea how does it work...

Specified xesmf version, added esmpy. Recent xesmf update buggy for regridding. See JiaweiZhuang/xESMF#47.

kthyng · 2020-09-28T19:28:32Z

I was able to get this to work with help from another issue.

I typically install ESMF/ESMPy with MPI support, else it does not work, properly, see here:

https://github.com/raphaeldussin/OM4p125_tideamp/blob/master/repro.yml

I used this yml as inspiration for what eventually worked for me, the important part being (not listing all the packages I use though):

name: env
channels:
  - conda-forge
dependencies:
  - esmf=8.0.1=mpi_openmpi_ha78a60a_0
  - esmpy=8.0.1=mpi_openmpi_py38h51f2404_0
  - pip:
    - git+https://github.com/pangeo-data/xESMF.git

Notes:

I figured out by guess and check how to change the esmf listing above by looking at the list of files here and changing it to match my mac operating system.
I didn't need to change anything for esmpy
This did not work using the conda-forge installation. I didn't test other options, but it worked by using the repository from pangeo-data listed above and installing with pip.

Pin xarray<=0.16.0

JiaweiZhuang closed this as completed Mar 14, 2019

JiaweiZhuang mentioned this issue Feb 28, 2020

ImportError: Regrid requires PIO #83

Closed

mathomp4 mentioned this issue Jul 29, 2020

xESMF 8 possible without MPI? #102

Open

shunzi-work mentioned this issue Aug 7, 2020

xesmf doesn't work in pangeo-notebook pangeo-data/pangeo-docker-images#101

Closed

tristanballard added a commit to tristanballard/pangeo_test that referenced this issue Aug 12, 2020

Update environment.yml

5268d18

Specified xesmf version, added esmpy. Recent xesmf update buggy for regridding. See JiaweiZhuang/xESMF#47.

LiamBindle added a commit to geoschem/gcpy that referenced this issue Aug 28, 2020

specified explicity esmf version, see JiaweiZhuang/xESMF#47 (comment)

c347811

bradyrx mentioned this issue Sep 29, 2020

Update CI to Github Actions pangeo-data/climpred#471

Merged

1 task

mickaellalande mentioned this issue Oct 21, 2020

Install instructions to avoid ImportError failing? #108

Closed

bradyrx mentioned this issue Oct 28, 2020

Use mamba + pytest-xdist to speed up CI testing pangeo-data/climpred#515

Merged

14 tasks

huard pushed a commit to Ouranosinc/xESMF that referenced this issue Nov 23, 2020

Merge pull request JiaweiZhuang#47 from pangeo-data/fix-36

ed3df2b

Pin xarray<=0.16.0

ashjbarnes mentioned this issue Sep 11, 2023

Installation doesn't (appear to) correctly set up XESMF COSIMA/regional-mom6#72

Closed

szwang1990 mentioned this issue Dec 11, 2023

pytest reports PIO errors #136

Open

hannahnesser pushed a commit to hannahnesser/gcpy that referenced this issue Jan 22, 2024

specified explicity esmf version, see JiaweiZhuang/xESMF#47 (comment)

088a7ea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support #47

ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support #47

yongjias commented Mar 13, 2019

JiaweiZhuang commented Mar 13, 2019 •

edited

Loading

yongjias commented Mar 13, 2019

JiaweiZhuang commented Mar 13, 2019 •

edited

Loading

yongjias commented Mar 13, 2019

yongjias commented Mar 14, 2019

JiaweiZhuang commented Mar 14, 2019

ks905383 commented Dec 5, 2019 •

edited

Loading

JiaweiZhuang commented Dec 5, 2019 •

edited

Loading

ks905383 commented Dec 6, 2019

bradyrx commented Dec 10, 2019

lastproxy commented Dec 30, 2019

matteodefelice commented Jan 23, 2020

JiaweiZhuang commented Jan 23, 2020

brianpm commented Jan 28, 2020

stefan-hofer commented Feb 5, 2020 •

edited

Loading

mickaellalande commented Mar 2, 2020

stefan-hofer commented Jun 15, 2020

bradyrx commented Jul 28, 2020

mickaellalande commented Jul 29, 2020

kthyng commented Sep 28, 2020 •

edited

Loading

ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support #47

ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support #47

Comments

yongjias commented Mar 13, 2019

JiaweiZhuang commented Mar 13, 2019 • edited Loading

yongjias commented Mar 13, 2019

JiaweiZhuang commented Mar 13, 2019 • edited Loading

yongjias commented Mar 13, 2019

yongjias commented Mar 14, 2019

JiaweiZhuang commented Mar 14, 2019

ks905383 commented Dec 5, 2019 • edited Loading

JiaweiZhuang commented Dec 5, 2019 • edited Loading

ks905383 commented Dec 6, 2019

bradyrx commented Dec 10, 2019

lastproxy commented Dec 30, 2019

matteodefelice commented Jan 23, 2020

JiaweiZhuang commented Jan 23, 2020

brianpm commented Jan 28, 2020

stefan-hofer commented Feb 5, 2020 • edited Loading

mickaellalande commented Mar 2, 2020

stefan-hofer commented Jun 15, 2020

bradyrx commented Jul 28, 2020

mickaellalande commented Jul 29, 2020

kthyng commented Sep 28, 2020 • edited Loading

JiaweiZhuang commented Mar 13, 2019 •

edited

Loading

JiaweiZhuang commented Mar 13, 2019 •

edited

Loading

ks905383 commented Dec 5, 2019 •

edited

Loading

JiaweiZhuang commented Dec 5, 2019 •

edited

Loading

stefan-hofer commented Feb 5, 2020 •

edited

Loading

kthyng commented Sep 28, 2020 •

edited

Loading