Skip to content

Roadmap

Jump to bottom
YadeWira edited this page Oct 10, 2026 · 11 revisions

Roadmap

State at v1.6.11-pre. Silesia L15: 25.12% at ~1,129 MB/s decode; zstd -19: 24.96% at ~807 MB/s (one idle pinned core, one session, 2026-10-09). lz6 now decodes 1.1-1.4x faster than zstd at the same ratio from level 2 up and still encodes slower (0.57-0.86x the speed, 3.8x slower at the top); zstd keeps the fast end (zstd -1 decodes 1.16x faster than lz6 -1) and lz6 leads where its transforms apply (see Benchmarks and bakeoff/SINGLE_THREAD_STUDY.md).

Done

  • Decoder rewrite 2, tight extra-bit widths, two memory-safety fixes (v1.6.11-pre): the seq decoder runs 143-163 instructions per sequence instead of 232-278 (level 3), decode cycles -34..-42% size-weighted over Silesia + AIT with byte-identical streams, no file slower; the length codes' extra-bit widths are log2 of the code step (format change, -0.2..-0.3% bytes, no file larger); an oversubscribed Huffman header and a block ending right after its literals no longer read or write out of bounds. See Decoder Internals and NEWS.

  • Seq v2 (v1.6.4-pre): tANS for ll/ml/of in one backward bitstream, one refill per sequence, bulk literal decode, prefetch 16 sequences ahead.

  • Entropy-priced optimal parser and a monotonic level ladder (v1.6.5-pre).

  • Offset low bits in the entropy coder (v1.6.6-pre): -1.1% on Silesia, mostly binary files.

  • Fast-level encode (v1.6.6-pre): L1-3 ~2.5x faster.

  • Decode stage A (v1.6.7-pre): rich 8-byte tANS entries and a branchless fill/steady/drain loop, -5% cycles.

  • Multithreaded command line (v1.6.10-pre): -T#, no format change; compression of the seq levels and decompression of every codec run blocks on an ordered pool, output identical to one thread. Silesia.tar at 16 threads: -9 5.0x, -5 3.9x, -3 3.6x, -1 2.3x; decode 445 -> 1,100-1,580 MB/s. See Benchmarks.

  • x86 branch filter, 32-slot rows, better plane lanes (v1.6.9-pre): ooffice 44.4% -> 38.2%, Silesia L15 25.46% -> 25.20%.

  • 4-stream Huffman literals, length-limited Huffman, single-pass L1-L2 (v1.6.8-pre): decode -10..-14% cycles, L1 174 / L2 144 MB/s encode.

  • Row-hash match finder for L3-L5 (v1.6.7-pre): same or better ratio than the chain parsers at up to 2x the encode speed; literal-mode choice made decode-aware (Huffman wins near-ties with order-1).

Decode speed

The structural gap of 2026-10-02 is closed (v1.6.11-pre). The measured gap to zstd was instruction count, not IPC: 230-290 instructions per sequence against zstd's 130-160, 71 of the 238 in the hot loop being stack loads and stores. The decoder was rewritten (no format change): the stage-B fast path with zstd's three tests, the last sequence peeled out of the loop, a negated-position bit reader with bzhi in the BMI2 clone, SSE2 plane merge, the Huffman 4-stream lockstep loop, a 64-sequence prefetch lead. Result at level 3 (xml / dickens / samba): 253 / 232 / 278 -> 163 / 143 / 163 instructions per sequence; decode cycles -38.2% / -40.9% / -38.1% / -37.8% at levels 1 / 3 / 8 / 15 (size-weighted, Silesia + AIT, no file slower). Details, what each step bought and what was tried and dropped: Decoder Internals.

What is left:

  • zstd -1 (34.55%) still decodes 1.16x faster than lz6 -1 (33.09%), and lizard -30/-49 decode 1.1-1.2x faster at their ratios.
  • Not tried: a NEON or other non-x86 specialisation (the plane merge has SSE2 and the sequence/Huffman loops BMI2 clones only; ARM64 runs the portable code, verified under qemu-user for correctness, never timed), an assembler decoder (uf-lzma2's assembler LZMA decoder gains 36-48% over its C one), a hand-scheduled stage A.
  • Dropped after measuring: forward LSB bitstream, unit-record tables, AVX2 lanes, hot/cold stage-A split, a run-time prefetch depth, the unconditional 2 x 16 B match copy, compiler flags and PGO (see Decoder Internals).

Ratio

  • The remaining Silesia losses to zstd -19 (lz6 -15 against zstd -19, v1.6.11-pre): samba (19.43 vs 18.04%), xml (9.34 vs 8.47%), mr (32.71 vs 31.16%), nci (5.33 vs 4.96%), mozilla (30.14 vs 29.41%), sao (69.90 vs 68.95%), reymont (20.86 vs 20.35%), dickens (28.29 vs 27.96%). lz6 is ahead on ooffice (38.15 vs 42.18%, the x86 filter), x-ray (55.80 vs 60.53%) and webster (20.61 vs 20.93%), and level with osdb (30.71 vs 30.74%).
  • Candidates: a literal model guided by the match byte (LZMA-style "matched literals" as a new coder, not a mode); a separate 4-bit align state for offsets (measured offline as close to the joint 3-bit symbol, so a small gain at best).
  • Refuted (see bakeoff/COMPETITORS.md and NEWS): order-1 gate changes, window caps, entropy prices in the L4-10 heuristic parsers, bucket "slots" for offsets.
  • Single block vs per file: LZ6S3 segmentation recovers only part of the ~5% gap on Silesia.tar.

Multithreading: what is left

  • Level 1 is bound by the main thread (~270 MB/s): reading, the LZ6S3 segmentation scan, XXH32 over the input, and the copy into the slot. Candidates: read straight into the slot buffers, hash on its own thread, histogram of the next window ahead of the cut decision.
  • The speed-up is bounded by the biggest block. The default B7 makes segments of tens of MB; splitting a block in two parallel halves would cost ratio, so for now the user picks -B4/-B5. A cheaper trade: cut earlier only when -T is set and say so.
  • --hc stays on one thread because LZ6HC_init keeps the hash table of the previous block (the bytes depend on the context that compresses a block). Clearing it would change the frozen HC output that zpaq-std vendors.

Encode speed

  • L1-L2 (single-pass, 144-174 MB/s): ~45% of the time is the parser, ~40% the entropy stage (histograms, Huffman and tANS writers). zstd -1 is still 2x faster. x-ray lost 0.47 points at L15 because its byte-plane lanes now use the level-1 parser: lanes could use their own setting.
  • The row hash saturates near L5's ratio with 16-slot rows; 32-slot rows (or a second, long-match hash) could take over L6-L7, which still use the chain parser (~15-20 MB/s).
  • The gap to zstd is larger than "zstd -1 is faster at a worse ratio": zstd -3 is smaller than lz6 -1/-2 at the same encode cost (Silesia 31.21% against 33.16%/31.65%), and zstd -6/-7 are ~1.3-1.7x cheaper to encode than lz6 -3/-4 at a similar ratio. Measured causes (one thread, cycles around the compress call): lz6 -1 costs 1.6-2.7x the cycles of zstd -1 (1.4-1.6x the instructions, IPC 1.5-1.9 against 2.1-2.7); its last-level-cache misses per input byte are ~100x zstd -1's (0.029 against 0.0003: lz6's hash tables are far bigger than zstd's level-1 table); and 8-10% of its time at level 1 (5.6% at level 3) is the kernel zeroing the fresh buffers of every call, which a context that reuses its workspace between blocks would avoid.

lz6

Clone this wiki locally