Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support decompression of compressed blocks of size ZSTD_BLOCKSIZE_MAX #3399

Merged
merged 1 commit into from
Dec 22, 2022

Conversation

Cyan4973
Copy link
Contributor

@Cyan4973 Cyan4973 commented Dec 22, 2022

The wording of the Zstandard specification
allows valid compressed blocks to be sized exactly ZSTD_BLOCKSIZE_MAX bytes (128 KB).
This generally does not happen, since an uncompressed block would have same size but suffer no decompression speed cost.

Starting with this patch, the libzstd decoder will now accept decoding a compressed block of size ZSTD_BLOCKSIZE_MAX exactly.

Note though that decoders from reference libzstd before < v1.5.4 consider this edge case as an error.

As a consequence, avoid generating compressed blocks of size ZSTD_BLOCKSIZE_MAX for broader compatibility with the ecosystem of deployed zstd decoders.
Note that the reference libzstd encoder never produces compressed blocks of size ZSTD_BLOCKSIZE_MAX, and will continue to do so for the foreseeable future, both to preserve compatibility with older decoders, and because it doesn't make sense to produce compressed blocks of such size (as opposed to uncompressed blocks).

The impact of this modification to the ecosystem is expected to be none. This is mostly meant to respect the letter of the RFC8878 specification.

Thanks to @SmitaKumar for reporting this issue in #2577

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants