New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZSTD compressor #93

Open
wants to merge 13 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@martint
Copy link
Member

martint commented Oct 4, 2018

No description provided.

@martint martint force-pushed the martint:zstd branch 2 times, most recently from b6c38e4 to 3aa3a05 Oct 4, 2018

@martint martint force-pushed the martint:zstd branch 2 times, most recently from 8490fd4 to bcf6f32 Nov 9, 2018

@martint martint force-pushed the martint:zstd branch from bcf6f32 to 4af80e7 Jan 11, 2019

@martint martint force-pushed the martint:zstd branch from 4af80e7 to 365d743 Jan 11, 2019

@martint martint changed the title WIP - ZSTD compressor ZSTD compressor Jan 11, 2019

@@ -90,4 +90,24 @@ public void testInvalidSequenceOffset()

getDecompressor().decompress(compressed, 0, compressed.length, output, 0, output.length);
}

@Test
public void testSmallLiteralsAfterIncompressibleLiterals()

This comment has been minimized.

@martint

martint Jan 11, 2019

Author Member

TODO: this test should move to the commit that adds huffman

sequenceCount = 0;
longLengthField = null;

// start = 0; // TODO for debugging

This comment has been minimized.

@martint

martint Jan 11, 2019

Author Member

TODO: remove

{
// TODO: store codes directly?

// if (start == 0) {

This comment has been minimized.

@martint

martint Jan 11, 2019

Author Member

TODO: remove commented out alternatives

@dain

This comment has been minimized.

Copy link
Member

dain commented Feb 7, 2019

These refactoring commits looks good:

Extract zstd constants to separate class  …
Rename weights -> outputBuffer
Rename size to capacity
Rename buildRleTable to initializeRleTable
Move FSE table load outside of decompress function
Update native zstd version
Add test with only ASCII and uniform distribution  …
Rename BitStream to BitInputStream
Add native zstd to compression benchmark
Make Util class final

Please merge these.

@dain
Copy link
Member

dain left a comment

One comment, but "Add no-op zstd compressor" looks good

@@ -58,7 +59,7 @@
xerial_snappy(new XerialSnappyDecompressor(), new XerialSnappyCompressor()),
iq80_snappy(new Iq80SnappyDecompressor(), new Iq80SnappyCompressor()),
hadoop_lzo(new HadoopLzoDecompressor(), new HadoopLzoCompressor()),
zstd_jni(new ZstdJniDecompressor(), new ZstdJniCompressor(6)),
zstd_jni(new ZstdJniDecompressor(), new ZstdJniCompressor(3)),

This comment has been minimized.

@dain

dain Feb 7, 2019

Member

You might want to do this one in another commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment