Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core dump or ignored exceptions on shutdown when reading from s3fs using fsspec #80

Closed
gerritholl opened this issue Sep 29, 2020 · 4 comments
Closed

core dump or ignored exceptions on shutdown when reading from s3fs using fsspec #80

gerritholl opened this issue Sep 29, 2020 · 4 comments

Comments

@gerritholl
Copy link

When I try to read a file opened using fsspec.open, I get a fatal Python error (core dump) on shutdown.

Using:

import fsspec, h5netcdf
with fsspec.open("s3://noaa-goes16/ABI-L1b-RadF/2019/321/14/OR_ABI-L1b-RadF-M6C01_G16_s20193211440283_e20193211449591_c20193211450047.nc", mode="rb", anon=True) as f:                                                                                                                   
    ds = h5netcdf.File(f)
    print(ds["Rad"][400, 300])

and then running with python -X dev gives:

/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/helpers.py:107: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
  def noop(*args, **kwargs):  # type: ignore
/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/connector.py:964: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  hosts = await asyncio.shield(self._resolve_host(
/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/locks.py:21: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  self._event = asyncio.Event(loop=loop)
1023                
Exception ignored in: <function File.close at 0x7f74bc830910>
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 802, in close
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 794, in flush
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 531, in _create_dim_scales
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 374, in _h5group
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5py/_hl/group.py", line 264, in __getitem__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
  File "h5py/h5i.pyx", line 43, in h5py.h5i.wrap_identifier
ImportError: sys.meta_path is None, Python is likely shutting down
Fatal Python error: PyThreadState_Get: no current thread
Python runtime state: finalizing (tstate=0x5621bb1ae670)

Aborted (core dumped)

when wrapping the opening in a context manager, the core dump is apparently averted, but there are still ignored exceptions reported on shutdown. Using:

import fsspec, h5netcdf
with fsspec.open("s3://noaa-goes16/ABI-L1b-RadF/2019/321/14/OR_ABI-L1b-RadF-M6C01_G16_s20193211440283_e20193211449591_c20193211450047.nc", mode="rb", anon=True) as f:                                                                                                                   
    with h5netcdf.File(f) as ds:
        print(ds["Rad"][400, 300])

gives

1023
Traceback (most recent call last):
  File "mwe90.py", line 4, in <module>
    print(ds["Rad"][400, 300])
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 811, in __exit__
    self.close()
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 803, in close
    self._h5file.close()
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5py/_hl/files.py", line 443, in close
    h5i.dec_ref(id_)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref
  File "h5py/defs.pyx", line 1234, in h5py.defs.H5Idec_ref
  File "h5py/h5fd.pyx", line 169, in h5py.h5fd.H5FD_fileobj_write
  File "h5py/h5fd.pyx", line 175, in h5py.h5fd.H5FD_fileobj_truncate
AttributeError: 'S3File' object has no attribute 'truncate'
Exception ignored in: <function File.close at 0x7ff52a33af70>
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 802, in close
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 794, in flush
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 531, in _create_dim_scales
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 374, in _h5group
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5py/_hl/group.py", line 264, in __getitem__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
ValueError: Not a location (invalid file ID)

It is puzzling that the traceback starts at the print line, despite the value being read out and printed to the console correctly.

I'm using fsspec 0.8.3, h5netcdf 0.8.1, netcdf4 1.5.4, s3fs 0.5.1, on Python 3.8.5 on openSUSE 15.0.
@gerritholl gerritholl mentioned this issue Sep 29, 2020
Closed
6 tasks
@gerritholl
Copy link
Author

When additionally I try to use fsspec blockcache I get additional tracebacks. Using

import fsspec, h5netcdf
of = fsspec.open(
    "blockcache::s3://noaa-goes16/ABI-L1b-RadF/2019/321/14/OR_ABI-L1b-RadF-M6C01_G16_s20193211440283_e20193211449591_c20193211450047.nc",
    target_protocol="s3",
    cache_storage="/data/gholl/cache/fscache/blockcache",
    s3={"anon": True},
    target_options={"anon": True})
with of as f:
    with h5netcdf.File(f) as hf:
        print(hf["Rad"][400, 300])

I get:

/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/helpers.py:107: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
  def noop(*args, **kwargs):  # type: ignore
/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/connector.py:964: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  hosts = await asyncio.shield(self._resolve_host(
/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/aiohttp/locks.py:21: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  self._event = asyncio.Event(loop=loop)
/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/caching.py:54: ResourceWarning: unclosed file <_io.BufferedRandom name='/tmp/tmpnm_ifnpp/12ebe5cb08aea5e1bc4f90fab4be1ddbe4a23972f4d183f4f829a2649ac48d0a'>
  self.cache = self._makefile()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
1023
Traceback (most recent call last):
  File "mwe91.py", line 10, in <module>
    print(hf["Rad"][400, 300])
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 811, in __exit__
    self.close()
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 803, in close
    self._h5file.close()
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5py/_hl/files.py", line 443, in close
    h5i.dec_ref(id_)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref
  File "h5py/defs.pyx", line 1234, in h5py.defs.H5Idec_ref
  File "h5py/h5fd.pyx", line 169, in h5py.h5fd.H5FD_fileobj_write
  File "h5py/h5fd.pyx", line 175, in h5py.h5fd.H5FD_fileobj_truncate
AttributeError: 'S3File' object has no attribute 'truncate'
Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7f87c8bfa5f0>
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/spec.py", line 1493, in __del__
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 338, in <lambda>
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 386, in <lambda>
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 354, in close_and_update
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 386, in <lambda>
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 152, in save_cache
NameError: name 'open' is not defined
Exception ignored in: <function File.close at 0x7f87c3f109b0>
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 802, in close
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 794, in flush
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 531, in _create_dim_scales
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5netcdf/core.py", line 374, in _h5group
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/h5py/_hl/group.py", line 264, in __getitem__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
ValueError: Not a location (invalid file ID)

That may be related to fsspec/filesystem_spec#404

@gerritholl gerritholl mentioned this issue Sep 29, 2020
Closed
@shoyer
Copy link
Collaborator

shoyer commented Oct 5, 2020

This sounds a lot like #50.

The summary of that issue is that almost any use of __del__ is a bug. You basically should avoid using __del__ in favor of weakref.finalize.

@kmuehlbauer
Copy link
Collaborator

@gerritholl I can't reproduce this with a fresh install. Could you please check again so that we might close this issue?

@kmuehlbauer
Copy link
Collaborator

This seems to be fixed in master now. At least, if mode='r' is used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gerritholl @shoyer @kmuehlbauer