Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native HDF5 backend #5

Open
svniemeijer opened this issue Dec 10, 2015 · 1 comment
Open

Native HDF5 backend #5

svniemeijer opened this issue Dec 10, 2015 · 1 comment

Comments

@svniemeijer
Copy link
Contributor

svniemeijer commented Dec 10, 2015

Instead of using the HDF5 library, create our own implementation to read HDF5 files.
This will allow a much faster access to the data (not need to work with dataspaces, vlen APIs, etc.)

The tricky part will be dealing with compressed data, but this should be similar to how we currently handle this with the CDF backend for zipped data.

@svniemeijer
Copy link
Contributor Author

Using partial array reads with the HDF5 backend currently imposes a penalty due to the use of H5Sselect_hyperslab(). This penalty can be considerable. There have been test cases where using a partial array read would actually be slower then just reading the full dataset (and then taking a slice from it in memory). Currently, a hyperslab selection may need to be much smaller than the overall size to actually see any performance gain.

An implementation of a native backend in CODA should also try to make sure that partial dataset reads will actually end up becoming faster than reading the full dataset (in most circumstances).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant