Repository navigation
Feature Request / Optimization: FSE-only Literal Compression Fallback for Constrained Alphabet Data (e.g., Base64/DNA) on Fast Levels #1162
unxed
started this conversation in
Ideas/Features
Replies: 1 comment
|
What you are proposing sounds a lot like WithAllLitEntropyCompression. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hello!
We are using
klauspost/compress/zstdin a high-performance cross-platform archiver. During our performance matrix benchmarks, we noticed an interesting behavior with highly literal-skewed but repeating-less datasets (such as a 50MB Base64-encoded binary stream).Observed Behavior:
zstd(Level 3): Compresses the 50MB Base64 file to~74.6%of its original size at218 MB/s. Since Base64 has exactly 64 unique bytes (6 bits of entropy per character), nativezstdbypasses sequence (LZ77) matching but still compresses the literals using FSE/Huffman coding.klauspost/compress/zstd(Default/Fast Levels): Bypasses compression completely, writing the data as uncompressed blocks at RAM speed (1012 MB/s), yielding exactly100.0%ratio.The Problem:
At default/fast levels, when the sequence search engine finds negligible or zero LZ77 matches in the block, the encoder completely gives up on entropy coding and writes fully uncompressed blocks to save CPU cycles.
Proposed Optimization / Heuristic:
Before deciding to write a block as completely uncompressed, the encoder could perform a lightweight check of the literal distribution (e.g., unique byte count or a fast entropy estimation):
Since ZSTD natively supports frames composed of literal-only blocks, implementing this fast-path fallback would drastically improve compression density for genomic (DNA), base64-encoded, and raw telemetry data streams without impacting the throughput on truly random streams.
We would love to hear your thoughts on introducing this heuristic to the fast-path encoder pipeline!
See also: klauspost/pgzip#70
All reactions