Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional block compression using LZ4 #32

Open
incubos opened this issue Jul 13, 2016 · 1 comment
Open

Optional block compression using LZ4 #32

incubos opened this issue Jul 13, 2016 · 1 comment

Comments

@incubos
Copy link
Contributor

incubos commented Jul 13, 2016

On write:

  • Fill a block until 64KB reached
  • Add a block header with payloads offsets
  • Compress the block using LZ4
  • Add a segment header with blocks offsets

On read (getDocumentById()):

  • Find the block using blocks offsets
  • Decompress the block into temporary buffer (possibly stopping after having decompressed enough bytes)
  • Slice the payload using payloads offsets
@incubos
Copy link
Contributor Author

incubos commented Oct 17, 2016

As an alternative we could use https://github.com/luben/zstd-jni thanks to support for dictionary compression (having trained dictionary using all the payloads).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant