Add libdeflate_deflate_decompress_stream() for streaming decompression - #2
Merged
Merged
Conversation
libdeflate's decompressor is one-shot: it decodes until BFINAL and cannot
resume on a partial stream, so it can't back a streaming decompressor for
arbitrarily large zlib/gzip/deflate input without buffering the whole thing.
Add a resumable variant that suspends at DEFLATE block boundaries. The only
cross-block state is {bitbuf, bitsleft} plus the 32 KiB history window, so
suspension is cheap: at each block boundary we checkpoint the byte-aligned
bit state, and if the input runs out (REFILL would overread and
end_of_input is false) or the output fills mid-block, we roll back to that
checkpoint and return STREAM_NEED_INPUT / STREAM_NEED_OUTPUT. The caller
supplies more input or drains output and calls again. Back-references are
resolved by having the caller place up to 32 KiB of previously produced
output immediately before 'out' and pass 'window_nbytes'; the match-offset
validation base is shifted accordingly.
The streaming code is a second instantiation of decompress_template.h guarded
by DEFLATE_STREAMING, so the existing one-shot decoder is byte-for-byte
unchanged (the only shared-macro change is a no-op OVERREAD_HANDLER() hook in
REFILL_BITS). A single block larger than the output buffer is handled by the
caller growing the buffer and retrying; mid-block suspension is intentionally
not supported.
Verified with 50000 randomized round-trip cases against zlib (random data
styles, sizes up to 3 MB, all levels, random input-chunk and output-region
sizes exercising both suspension paths and the window carry-over).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
libdeflate_deflate_decompress_stream()(+_reset()): a resumable raw-DEFLATE decoder that suspends at block boundaries, so arbitrarily large zlib/gzip/deflate streams can be decompressed with bounded memory using libdeflate's fast table-driven decoder.Why
libdeflate's decompressor decodes untilBFINALand cannot resume on a partial stream, so it cannot back a streaming decompressor (e.g. reading large.gzfiles or HTTP bodies) without buffering the entire input. This lets ClickHouse route streaming gzip/zlib decompression through libdeflate (~1.4–1.5× faster than zlib-ng in our benchmarks) instead of keeping a separate streaming codec.How
{bitbuf, bitsleft}+ the 32 KiB window. At each block boundary we checkpoint the byte-aligned bit state. If the input runs out mid-block (aREFILL_BITSoverread whileend_of_inputis false) or the output fills, we roll back to the checkpoint and returnLIBDEFLATE_STREAM_NEED_INPUT/LIBDEFLATE_STREAM_NEED_OUTPUT.outand passeswindow_nbytes; the match-offset validation base is shifted bywindow_nbytes. (DEFLATE offsets are ≤ 32 KiB, so this resolves every back-reference with no per-reference special-casing.)decompress_template.hguarded byDEFLATE_STREAMING. The existing one-shot decoder is byte-for-byte unchanged; the only shared-macro change is a no-opOVERREAD_HANDLER()hook inREFILL_BITS.Limitation
Suspension is at block boundaries only. A single block larger than the output buffer is handled by the caller growing the buffer and retrying (the decoder re-decodes from the block start). Mid-block suspension is intentionally not implemented (it would roughly reimplement zlib's inflate state machine in libdeflate's hot loop).
Testing
50,000 randomized round-trip cases against zlib: random data styles (incompressible, low-entropy, text, RLE-runs), sizes up to 3 MB, all compression levels, and random input-chunk and output-region sizes that exercise both suspension paths, the 32 KiB window carry-over, and oversized-block grow-and-retry. All passed.
Based on
v1.25-base(= v1.25 + the merged streaming-compress PR), so the diff is exactly this addition.🤖 Generated with Claude Code