Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_hdf closes HDF5 stores that it didn't open. #28699

Closed
kboone opened this issue Sep 30, 2019 · 1 comment · Fixed by #28700
Closed

read_hdf closes HDF5 stores that it didn't open. #28699

kboone opened this issue Sep 30, 2019 · 1 comment · Fixed by #28700
Labels
Bug IO HDF5 read_hdf, HDFStore
Milestone

Comments

@kboone
Copy link
Contributor

kboone commented Sep 30, 2019

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.DataFrame({'a': range(10), 'b': range(10)})

df.to_hdf('/tmp/1.hdf', 'key2')

store = pd.HDFStore('/tmp/1.hdf', 'r')

try:
    pd.read_hdf(store, 'key1')
except KeyError:
    pass

pd.read_hdf(store, 'key2')

Problem description

Due to a recent change in pandas, the read_hdf function closes HDF5 stores if it fails to read from the file. In my opinion, this is the expected behavior if read_hdf opened the file itself, but shouldn't happen if it was passed a file that is already open. One use case for this is a series of try/except blocks that try to read a series of different keys from an HDF5 file that may or may not have the individual keys, as in the example above. The attempt to read key2 will fail because the first attempt will have closed the file.

This issue was introduced in pull #25863 to address issue #25766 where read_hdf wasn't closing files that it hadn't opened itself.

The fix is pretty easy, just add if not isinstance(path_or_buf, HDFStore): before store.close() at the end of read_hdf().

Expected Output

Should not raise an Exception when reading key2.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-957.27.2.el7.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.utf8
LANG : en_US.utf8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.2
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.3
setuptools : 41.0.1
Cython : 0.29.13
pytest : 5.1.2
hypothesis : None
sphinx : 2.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.8.0
pandas_datareader: None
bs4 : 4.8.0
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : 2.7.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : 3.5.2
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

@kboone
Copy link
Contributor Author

kboone commented Oct 1, 2019

I submitted PR #28700 to fix this.

@jreback jreback added Bug IO HDF5 read_hdf, HDFStore labels Oct 3, 2019
@jreback jreback added this to the 1.0 milestone Oct 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug IO HDF5 read_hdf, HDFStore
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants