ImportError: sys.meta_path is None, Python is likely shutting down

[floriankrb]

This works :

import xarray as xr    
ds = xr.open_mfdataset(['test.h5'], concat_dim = 'datetime', engine='h5netcdf',decode_cf=False)                                                                                     
ds_new = ds['var'].copy()  

But this does not :

import xarray as xr                                                                                                                                                                 
def f():                                                                                                                                                                            
   ds = xr.open_mfdataset(['test.h5'], concat_dim = 'datetime', engine='h5netcdf',decode_cf=False)                                                                                 
   ds_new = ds['var'].copy()                                                                                                                                                       
   return ds_new                                                                                                                                                                   
ds_new = f()     

The following exception message is witten on stderr when the script ends.

Exception ignored in: <bound method CachingFileManager.del of CachingFileManager(<function open_h5netcdf_group at 0x7f0a90e7ec80>, '/home/pinaultf/sandbox/dev/xarray/test.h5', mode='r', kwargs={'group': None})>
Traceback (most recent call last):
File "/home/pinaultf/sandbox/dev/xarray/xarray/backends/file_manager.py", line 210, in del
File "/home/pinaultf/sandbox/dev/xarray/xarray/backends/file_manager.py", line 188, in close
File "/home/pinaultf/sandbox/dev/xarray/xarray/backends/netCDF4.py", line 241, in close
File "/home/pinaultf/miniconda3/envs/env_vegeo_12_2018/lib/python3.6/site-packages/h5netcdf/core.py", line 665, in close
File "/home/pinaultf/miniconda3/envs/env_vegeo_12_2018/lib/python3.6/site-packages/h5py/_hl/files.py", line 330, in close
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 236, in h5py.h5f.get_obj_ids
File "h5py/h5i.pyx", line 43, in h5py.h5i.wrap_identifier
ImportError: sys.meta_path is None, Python is likely shutting down

As far as I kow, this may be related to #21 and to the garbage collector.

Note that adding "ds.close()" removes the message.

import xarray as xr                                                                                                                                                                 
def f():                                                                                                                                                                            
   ds = xr.open_mfdataset(['test.h5'], concat_dim = 'datetime', engine='h5netcdf',decode_cf=False) 
   ds.close()
   ds_new = ds['var'].copy()                                                                                                                                                       
   return ds_new                                                                                                                                                                   
ds_new = f()     

Works as expected.

[floriankrb]

xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.35-server-1.mga6
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8

xarray: 0.11.0
pandas: 0.23.4
numpy: 1.15.4
scipy: 1.1.0
netCDF4: 1.4.2
h5netcdf: 0.6.2
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.0.2.1
PseudonetCDF: None
rasterio: None
iris: None
bottleneck: 1.2.1
cyordereddict: None
dask: 1.0.0
distributed: 1.25.0
matplotlib: 3.0.2
cartopy: 0.17.0
seaborn: 0.9.0
setuptools: 40.6.2
pip: 18.1
conda: None
pytest: 3.10.1
IPython: 7.2.0
sphinx: 1.8.2

[shoyer]

Could you share a fully reproducible example, including a copy of the netcdf file or code to create it?

[shoyer]

I would also suggest trying out xarray 0.11.2, which includes some fixes related to this functionality.

[DerWeh]

Is this still active? I do get the same error:

Exception ignored in: <function CachingFileManager.__del__ at 0x7f8625ebcc80>
Traceback (most recent call last):
  File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 243, in __del__
  File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 221, in close
  File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/h5netcdf/core.py", line 701, in close
  File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/h5py/_hl/files.py", line 431, in close
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 267, in h5py.h5f.get_obj_ids
  File "h5py/h5i.pyx", line 43, in h5py.h5i.wrap_identifier
ImportError: sys.meta_path is None, Python is likely shutting down

Using

lay_dat = xr.open_mfdataset(files.values(), engine='h5netcdf', concat_dim=Dim.it, combine='nested')

A within a function always raises the error, using result = xr.open_dataset(output, engine='h5netcdf') this error only occurs if I do not manually invoke result.close().

---

I have to my knowledge the most recent versions installed:

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.16.0-10-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: None

xarray: 0.14.1
pandas: 0.25.0
numpy: 1.16.4
scipy: 1.3.0
netCDF4: None
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.3.0
distributed: 2.3.0
matplotlib: 3.0.3
cartopy: None
seaborn: None
numbagg: None
setuptools: 41.0.1
pip: 19.1.1
conda: 4.7.11
pytest: 5.0.1
IPython: 7.5.0
sphinx: 2.2.1

[shoyer]

I now think this is a probably a bug on the xarray side. We should be automatically cleaning-up unclosed files using atexit rather than only relying on __del__: https://stackoverflow.com/questions/14986568/python-how-to-ensure-that-del-method-of-an-object-is-called-before-the-mod

If you could share a script that reproduces the bug that would be very helpful for testing. Please file an issue in the xarray tracker and ping me. Thanks!

[DerWeh]

Thanks for the info, I will try to come up with a minimal script this weekend and open a pull request.

I see. Using __del__ is indeed no reasonable way to clean up. Before switching to xarray I had a hacky solution to automatically close my files, which I happily appended and deferred the problem to xarray. I patch something together using weakref.finalize, which sounded to me like the right solution. weakref.finalize is called when the object is garbage collected or atexit.

I used a wrapper around the file object, having the following method:

 def _autoclean_load(self, *args, **kwds):
      data = np.load(*args, **kwds)
      finalize(self, data.close)
      return data

If the wrapper object is collected or the script ends in a way, such that atexit is applied, data.close is called. If I remember correctly you need the wrapper class, as finalize contains the reference to data.close, thus data would only be collect on exit.

See my question on stack overflow

[shoyer]

It sounds like you may have looked into this in more detail than I have!

Take a look here for the logic in xarray. Pull requests to improve this would be very welcome!

[mdgoldberg]

Hey, I'm still running into this issue using xarray. My versions are xarray 0.15.0 and h5netcdf 0.7.4.

It seems that adding dataset.close() doesn't fix the issue unless you also load()/compute() the dataset. My process is open_mfdataset followed by close(), but I still get a long list of those h5py messages (one for each file opened by open_mfdataset).

@shoyer did this issue ever make it to the xarray issue tracker? I couldn't find it by searching. If it has could you reference that issue here so others who stumble upon it here can follow along there? This is the first issue that comes up when I google that ImportError: sys.meta_path is None, Python is likely shutting down error message.

[shoyer]

I don't think has made it onto the xarray issue tracker yet. It would still be helpful to have a full script that reproduces the behavior for debugging potential fixes.

[kmuehlbauer]

@shoyer Will this be resolved with XARRAY_BACKEND_API = v2? If we had a MWE, we could add tests for this.

[kmuehlbauer]

@floriankrb I can't reproduce this with h5netcdf version 0.12.0. If you still encounter this issue, please provide a minimal working example showing this behaviour. We can reopen then. Thanks.

[floriankrb]

I don't have the original file anymore and I cannot reproduce either.

Thank you for closing this issue.