Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bitshuffle Filter #366

Closed
Heiney-Upenn opened this issue Apr 3, 2022 · 4 comments
Closed

Bitshuffle Filter #366

Heiney-Upenn opened this issue Apr 3, 2022 · 4 comments
Assignees
@jamesmudd
Labels
enhancement New feature or request

Comments

@Heiney-Upenn
Copy link

Heiney-Upenn commented Apr 3, 2022 •
edited

jhdf is currently missing the bitshuffle filter. When I try to open the primary dataset in a the following hdf file HDF5 file
12_ID_B_Sagbe_2M_SAXS_Eiger9M.h5.zip
with compressed data I get

Exception in thread "main" io.jhdf.exceptions.HdfFilterException: A required filter is not available: name='bitshuffle; see [https://github.com/kiyo-masui/bitshuffle' id=32008](https://urldefense.com/v3/https:/github.com/kiyo-masui/bitshuffle'*20id=32008;JQ!!IBzWLUs!F
8uUItD4EWpvvhym4ZgnK6-Dekob-TlEE18ADfKCnY4IXw1KVlXMpdTGZ0oS8QVg3g$)

at io.jhdf.filter.FilterManager.getPipeline(FilterManager.java:90)

It would be great if this could be implemented.
@Heiney-Upenn Heiney-Upenn changed the title suggestion Bitshuffle Filter Apr 3, 2022
@jamesmudd jamesmudd added the enhancement New feature or request label Apr 5, 2022
@jamesmudd
Copy link
Owner

Some work towards this on #370 still lots to do though

@jamesmudd
Copy link
Owner

I have made lots of progress on this code is updated on the PR. There are still a few more things to figure out, and want to restructure the code a bit around filters but, here is a beta version of the library that can open bit shuffled datasets with no or LZ4 compression (no zstd yet) jhdf-0.6.6-bitshufle-beta.zip (.zip needs renaming to .jar for github)

@jamesmudd jamesmudd self-assigned this May 5, 2022
@jamesmudd
Copy link
Owner

Missed something in the previous version hopefully this is better. jhdf-0.6.6-bitshufle-beta2.zip

@jamesmudd
Copy link
Owner

Merged the changes for this here #370 will be in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jamesmudd jamesmudd

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jamesmudd @Heiney-Upenn