Findings

Findings

What was measured while working on uf-lzma2, including the ideas the measurements ruled out, so that nobody has to rediscover them. Each result is from Silesia on the bakeoff machine (2x Xeon E5-2697A v4) unless stated otherwise.

Decompression

The assembler decoder is the big win, and lzbench did not have it. lzbench compiles fast-lzma2 from its C files only. Adding the same library with Igor Pavlov's assembler decoder: +36% (L10), +37% (L5), +48% (L1), same compressed bytes.

Encoder parameters barely move decode speed. Level 5, one parameter at a time, each stream decoded in round-robin passes on a quiet machine (baseline 114.3 MB/s, noise about 1%):

parameter	effect on decompression
lc, lp, pb	within about 2%
fastLength, searchDepth	within about 2%
strategy fast	+2.6%, but ratio 1.1 points worse
dictionary 1 MB → 64 MB	106.5 → 118.3 MB/s, and ratio 25.32 → 23.68%

The threshold set before measuring was 15-20% to justify an encoder that favours fast-to-decode parses. The result was about 2%, so it was not built. A larger dictionary helps both axes at once, which contradicts the expectation that far copies would be slower: more history means more matches, fewer symbols to decode, and that wins. It stops helping once the dictionary covers the file.

The .xz check. A byte-at-a-time CRC64 cost 22% of decode time; slicing-by-8, 6.1%; slicing-by-16, 4.6%. See Output formats.

Ratio

A "level 11" is not worth it. Level 10 already uses the maximum search depth and fast length, the ultra strategy, and no divide-and-conquer. What remains:

on top of level 10	Silesia ratio
level 10	22.9657%
hybrid cycles 64 (level 10 uses 8)	22.9446%
high-compression table, level 10	22.9470%
overlap fraction 4, 8, 14	22.9657% (no effect: each file fits in one block)

Choosing lc/lp/pb per file is the real ratio lever; choosing one for everyone is not. All 60 legal combinations at level 10, the best one per file, in size:

corpus	best of 60 per file	{3/0/2, 4/0/1, 1/2/2} per file
Silesia	-0.54%	-0.39%
AIT A-H	-2.35%	-2.16%
52 varied files	-0.44%	-0.28%
80 held-out files		-1.06%

The three-candidate set was chosen to do well on all three corpora at once, then checked on 80 files not used to choose it. Leaving the default 3/0/2 out of the candidates gave nearly the same total but made 32 of those 80 files larger than today; with it in, none can be. This is level 11.

A single better default does not exist. 3/1/3 is the best one for Silesia (-0.045 points) and good on AIT (-0.63), and it made 51 of the 52 varied files larger. lp and pb help aligned numeric data and hurt the rest; each file has to keep what suits it.

Choosing per chunk instead of per file does not pay. LZMA2 can carry new properties in any chunk, but only together with a state reset, so every switch restarts the probability model. The encoder already starts each thread's slice with such a reset, which made the question measurable without new code: encode with N slices and every one of the 60 settings, then take the best setting for each slice. The combined stream is valid LZMA2.

Silesia	best of 3 per slice	best of 60 per slice
1 slice, per file (level 11)	22.8760%
4 slices	22.8946%	22.8623%
16 slices	22.9435%	22.9050%
64 slices	23.0619%	23.0124%

AIT behaves the same way: level 11's 52.64% is beaten only at 4 slices with all 60 settings, and by 0.009 points. The resets cost more than the choice gains, and since the format ties a properties change to a state reset, that cost is intrinsic.

The reset cost is also the price of multithreaded compression, because each thread encodes one slice. Level 10 on Silesia with a fixed setting: 4 threads +0.08% in size, 16 threads +0.27%, 64 threads +0.77%; on AIT +0.11%, +0.80% and +0.92%.

Why not break compatibility

Decompression speed is the axis LZMA2 cannot win: its range coder decodes one adaptive bit at a time. To see whether leaving the format would pay, the exact parse of uf-lzma2 L10 was extracted (an instrumented copy of the C decoder, output verified identical) and its size estimated under fast static-table entropy coders of the kind zstd and brotli decode quickly:

same parse, coded as	Silesia
LZMA2 (actual)	22.97%
order-0 literals, zstd-style sequences	25.86%
64 literal contexts, matched literal, modelled low offset bits	25.43%

Robust to the table block size (128 KiB: 25.48%, 4 MiB: 25.50%). The obvious suspect, LZMA's many 1-byte copies (26.5% of copies), turned out not to be it: turning short copies into literals makes it worse (25.74% for length 1, 26.13% up to length 3).

On the same parse, LZMA's adaptive modelling is worth about 2.5 points, and it is also the source of the decode ceiling. A new format with static tables would land between zstd (24.96%) and brotli (23.75%), where zstd, brotli and lz6 already are. The one route not measured is keeping the adaptive model and decoding several streams in parallel. The conclusion was to stay compatible.

AIT

lz6's 7-point lead on AIT is one file: D, 2 MB of pseudo-random output that lz6 stores as the seed in 7 bytes. Without D uf-lzma2 L10 is at 45.50% against lz6's 46.66%. lz6 still wins E and F, numeric data it splits into byte planes; the standard xz delta filter makes both larger at every distance (E 85.97% → at best 88.99%, F 82.37% → at best 83.72%), so there is no compatible route there yet.

The same investigation found that lz6's result parser rebuilt each file's original size from the rounded ratio, so D, at ratio 0.00, dropped out of the denominator and lz6's AIT result was understated (46.66% reported, 39.55% real). lz6 fixed it in 12d8fe5.

How to measure without fooling yourself

Every one of these produced a wrong number at least once during this work.

• Small files measure overhead. 81 KB average: 65 MB/s decompression; one 29 MB file: 92 MB/s.
• Build like with like. A C-decoder build of one codec against an assembler build of another credits the assembler to the wrong design.
• Per file caps the dictionary at the largest file; use a solid stream for dictionary questions.
• Best of one is noise. Two runs of identical code differed by 10% at -n1.
• Shared machines interfere. A neighbour's 24 parallel compressions turned +1.9% into -17%. Decode every candidate in round-robin passes, keep the best of each, and measure only when the machine is quiet.
• An exit code can belong to the wrong process. cmd | tail reports tail's status; pkill -f and pgrep -f match their own command line.
• An empty count can be a missing file. Check that the thing counted exists before trusting a zero.