perf: compression optimizations (match extension, huffman decode, xz/lzma2 ratio) - #112
Merged
Merged
Conversation
Replace the byte-at-a-time forward match-extension loops in six LZ encoders with an 8-byte little-endian u64 XOR + trailing_zeros scan (match_forward), the same technique the deflate/zstd/brotli extenders already use. Emitted match lengths — and therefore the compressed bitstreams — are byte-for-byte identical; only the scan speed changes. Encode throughput on 1 MiB inputs (min MB/s, this host): codec lorem before->after source before->after lz4 1773 -> 13950 (7.9x) 418 -> 828 (2.0x) xpress 3125 -> 9998 (3.2x) — 538 lzo 2453 -> 7944 (3.2x) 470 -> 636 (1.35x) snappy 1758 -> 2989 (1.7x) 406 -> 605 (1.5x) lzs 902 -> 1182 (1.31x) — 216 adc 635 -> 878 (1.38x) — 305 lznt1 is intentionally left byte-at-a-time: its matches self-overlap (prev_pos + (len % dist)), which a word-wide compare cannot model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The standalone Huffman codec's decode loop rebuilt an 8-byte big-endian bit window from memory on every symbol (BitReader::peek's 8-iteration byte loop) and read two parallel tables (sym_tbl, len_tbl) per lookup. Replace both: - Maintain a left-aligned 64-bit accumulator refilled 8 bytes at a time, so the per-symbol cost is a shift + a single table load instead of a from-scratch window assembly. The tail still reads as zero-padded, matching the old peek exactly, and the truncation / corrupt-table checks are preserved. - Pack the two tables into one Vec<u16> ((len << 8) | sym) so a lookup touches a single cache line. Decoded output is unchanged. Decode throughput on 1 MiB: source 180 -> 271 MB/s (+51%), lorem 181 -> 375 MB/s (+107%). BitReader is now unused and removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- examples/micro.rs: a subprocess-free single-codec micro-benchmark (min/median MB/s over N runs with a warmup) for clean before/after measurement of one algorithm + input, complementing the full reference-comparison harness in examples/bench.rs. - BENCH.md: regenerated on this host (Intel i9-14900K). Reflects the word-at-a-time match-extension speedups (lz4 Lorem encode ~1.8 -> 14.9 GB/s, etc.) and the current bounded-memory lzma/xz encode tradeoff. Analysis prose rewritten to match the new numbers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ressible) The bounded-memory streaming LZMA2 encoder reset the whole probability model, state, and rep distances at the start of every 64 KiB chunk (control 0xC0), so the adaptive model never warmed past one chunk. On highly compressible input this bloated output badly: 16 MiB of Lorem encoded to ~21.8 KB vs native xz's ~2.9 KB (~7.5x larger). Continue the model across chunks instead, matching native xz: - Continuation compressed chunks now frame as control 0x80 (no state / props reset, no props byte); the first chunk stays 0xE0 and a chunk following an uncompressed-fallback chunk stays 0xC0. - The encoder core resets only the per-chunk range coder between continued chunks (new `reset_range_coder`), keeping probs / state / reps / output_pos. `Lzma2Chunk` carries a `reset_state` flag; the stream encoder tracks `need_state_reset` and forces a reset after any uncompressed chunk (whose discarded compressed attempt mutated the model the decoder never sees). The decoder already handled all four reset modes, so this is encoder-only. Verified byte-identical round-trips through our decoder AND decode by native `xz -d` on many-chunk compressible and incompressible inputs (strengthened tests/xz.rs; also fixed its pipe_through helper to stream stdin on a thread so large payloads don't deadlock). Result (16 MiB Lorem): xz 21872 -> 5552 bytes, lzma2 21815 -> 5497 bytes (~3.9x smaller; ~7.5x -> ~1.9x vs native xz). Incompressible and realistic mixed data are unchanged in size. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xz Lorem output 1708 -> 752 B, Zeros 1368 -> 440 B after the LZMA2 continuation-chunk model carry; note the effect and the closed gap to native xz in the analysis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ratio) The streaming encoder emitted a fresh LZMA2 chunk every 64 KiB, so the per-chunk framing header + 5-byte range-coder flush (~10 bytes/chunk) was paid 256 times over a 16 MiB stream — ~2.5 KB of pure overhead that dominated the remaining ratio gap vs native xz after the model-carry fix. Pack up to CHUNK_UNCOMP_MAX (1 MiB) of uncompressed data into each compressed chunk. Because a chunk's compressed body must fit the 16-bit comp_size field, encode_greedy/encode_optimal now take a comp_cap and return the position they actually reached, cutting the chunk at a clean packet/window boundary once the body reaches CHUNK_COMP_CAP (58 000 — leaves margin for the last committed window + flush under 65 536). So a chunk ends at whichever comes first: 1 MiB uncompressed, the compressed cap, or end of input. Uncompressed-fallback chunks still fit the 16-bit uncomp field (a fallback implies uncomp_len <= body <= the cap). The compressed-chunk uncompressed-size field is 21 bits (2 MiB), so the framing already supported this; only the per-chunk size cap and the framing debug-asserts changed. Encoder-only; decoder unchanged. Result (16 MiB Lorem): xz 5552 -> 2976 B, lzma2 5497 -> 2921 B — now ~1.03x native xz (was ~1.9x). 1 MiB Lorem: xz 752 -> 588, Zeros 440 -> 280. Incompressible unchanged. Encode speed unchanged. Verified round-trip + native `xz -d` (tests/xz.rs many-chunk cases now span 1 MiB chunks) + debug_assert invariants under medium-entropy stress. Tradeoff: the streaming buffer floor rises from ~64 KiB to ~1 MiB (memory stays O(dict_size + 1 MiB), bounded and input-length-independent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xz Lorem output 752 -> 588 B, Zeros 440 -> 280 B; 16 MiB Lorem now ~3.0 KB (~1.03x native xz). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- rustfmt examples/micro.rs - huffman_codec: drop the orphaned BitReader doc line left after removing the reader; drop a redundant `u16` cast - snappy: reorder match_forward above compress_block so compress_block's doc comment attaches to it again (fixes doc_lazy_continuation) No functional change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`examples/micro.rs` uses `compcol::factory`, so it must not be built when `factory` is off. The narrow-feature-subset clippy job builds `--all-targets` with `--no-default-features --features lz4`, which tried to compile the example without `factory` and failed. Declare `required-features = ["factory"]` for it, mirroring the `bench` example.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A sweep of profiled, benchmarked optimizations. Every change keeps output byte-identical / spec-correct; verified by the full test suite (1706 tests) plus native
xz -dcross-decode for the LZMA2/xz work.Throughput / ratio wins
Word-at-a-time forward match extension (
91eb503) — replaced byte-at-a-time match-extension loops with an 8-byte LEu64XOR +trailing_zerosscan (the technique deflate/zstd/brotli already used) in lz4, snappy, lzo, xpress, lzs, adc. Encode on 1 MiB Lorem: lz4 1773 → 13950 MB/s (7.9×), lzo 3.2×, xpress 3.2×, snappy 1.7×, lzs/adc ~1.3×. Output byte-identical.huffman_codec decode (
9cfd96a) — replaced the per-symbol 8-byte window rebuild with a maintained 64-bit accumulator, and packed the two parallel decode tables into oneVec<u16>. Decode +51% (source) to +107% (lorem).LZMA2/xz compressible ratio (
4b28e97,1483d65) — two fixes closing the gap to nativexz:0x80) instead of resetting every 64 KiB.Result — 16 MiB Lorem
xz: 21,872 → 2,976 bytes (~1.03× native xz, was ~7.5×). Incompressible size and encode speed unchanged. Tradeoff: streaming buffer floor rises from ~64 KiB to ~1 MiB (memory staysO(dict_size + 1 MiB)).Tooling / docs
examples/micro.rs— a subprocess-free single-codec micro-benchmark used for the before/after numbers.BENCH.mdregenerated on the current host;tests/xz.rsstrengthened with many-chunk native cross-decode cases (and itspipe_throughhelper fixed to stream stdin on a thread).🤖 Generated with Claude Code