-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zstd streaming and Hadoop Codec #152
Conversation
2decb38
to
e6c2f37
Compare
This appears to resolve #112 |
I'd love to see this merged and released; we're holding off on evaluating this library until the streaming API is available. |
throws IOException | ||
{ | ||
XxHash64 hash = new XxHash64(seed); | ||
byte[] buffer = new byte[8192]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why 8k?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I coppied this from Slice https://github.com/airlift/slice/blob/master/src/main/java/io/airlift/slice/XxHash64.java#L179
src/main/java/io/airlift/compress/zstd/ZstdFrameDecompressor.java
Outdated
Show resolved
Hide resolved
src/main/java/io/airlift/compress/zstd/ZstdFrameDecompressor.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rewrote the decompressor into a standalone, fully encapsulated, class, ZstdIncrementalFrameDecompressor
throws IOException | ||
{ | ||
XxHash64 hash = new XxHash64(seed); | ||
byte[] buffer = new byte[8192]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I coppied this from Slice https://github.com/airlift/slice/blob/master/src/main/java/io/airlift/slice/XxHash64.java#L179
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These commits look good:
- Add streaming XxHash64
- Rename parameter to be more descriptive
- Support unknown input size in ZSTD CompressionParameters
- Add ZstdInputStream
- Add window size and block size to CompressionParameters
Resolve #112