Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File open from file-object #687

Closed
Kriechi opened this issue Mar 10, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@Kriechi
Copy link

commented Mar 10, 2016

Right now h5py can only open files by passing a file path to it.
I'm working with compressed files - so my HDF5 file is in a xz container with lzma2 compression.

Ideally I would like to do something like this:

import lzma
import h5py

input_file = lzma.open('foobar.hdf5.xz', 'r')
f = h5py.File(input_file)
@aragilar

This comment has been minimized.

Copy link
Contributor

commented Mar 10, 2016

It appears to be a limitation of the HDF5 library: if you look at how h5py opens files, you'll see that it does very little with arguments you pass it; instead, it passes the arguments to the HDF5 library, which performs the low level work (getting a file handle etc.). If support for accepting existing file handles was added to the HDF5 library, then there shouldn't be much difficulty adding it to h5py.

@andrewcollette

This comment has been minimized.

Copy link
Member

commented Mar 14, 2016

@aragilar is right on this... unfortunately we get to pass the file name to HDF5 and not much else. There is some provision for working with buffers and in-memory files, but nothing I'm aware of that will take a Python file-like object. Would be cool, though!

If you're interested in using compression, the "standard" HDF5 way is to use compressed datasets. These are where 99% of the bytes in the file live, so it's surprisingly effective.

http://docs.h5py.org/en/latest/high/dataset.html#filter-pipeline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.