Development

Development

Corpora

• AIT A-H (13.1MB, 8 files): the training set the project was tuned against. It is kept in bakeoff/corpora/ and restored to /tmp/ait_train by bakeoff/setup_competitors.sh.
• Silesia (212MB, 12 files): external, never used for tuning. AIT alone flatters lz6, so both corpora are reported.

The bakeoff/ harness

script	what it does
bench_seq_speed.sh [--level N] [--asan] [--save F]	seq round-trip + enc/dec MB/s on AIT, into exact-size buffers
bench_seq_speed.sh --asan --fuzz N	ASAN fuzz of corrupted raw seq streams
bench_seq_speed.sh --asan --ffuzz N	ASAN fuzz of corrupted frames through LZ6F_decompress
bench_decode.sh / bench_encode.sh	HC-codec decode/encode gates (round-trip, ASAN fuzz, byte-exact ratio vs encode_baseline/)
run_vs_codecs.sh setup|ait|silesia	lzbench head-to-head with lz6 + lz5 as plugins (lzbench_lz6.patch)
parse_lzbench.py	lzbench dumps → the tables in VS_CODECS.md
mk_legacy_fixture.c	rebuilds the LZ6S1 compatibility fixture
seqgate.sh [REFTREE]	the standing safety gate of the seq decoder (both clones, exact-size buffers, ASAN + UBSan, mutation fuzz, differential run against REFTREE, 32-bit build); GATE_N=300000 TINY_N=5000 for a finished change; XDIFF_EXT=1 for a backward-compatible format extension

Other test programs live in programs/: frametest, seqframetest (seq frames, streaming, odd dst slices) and lz6seq_test.

Rules for performance work

These rules come from bakeoff/TASK.md and past mistakes:

1. Round-trip is a hard gate. A faster wrong decoder is a failure.
2. Safe on malformed input. Never widen a copy that runs before its bounds check. Fuzz into exact-size output buffers, because over-allocated buffers (like the CLI's) hide out-of-bounds writes.
3. Measure against the right baseline. Build the baseline from git archive master and run base and candidate interleaved on a pinned core. This machine is shared: background load moves wall-clock numbers by ±5% and even cycle counts by ±3%. An early baseline in the decode work was taken while 24 compressions ran in the background and looked 2x slower than it was.
4. Verify the consequence, not only the fix. Check what else the change touches. Example: capping FSE precision is a decoder win, but it also changes the encoder's output, so ratio is re-measured on both corpora.
5. Portable scalar path. No SIMD-only code. Compiler builtins go behind #if defined(__GNUC__) with a fallback.
6. Check the denominator. Aggregates must use the real file sizes, not sizes rebuilt from a rounded ratio. A 0.00% ratio (lz6 on the PRNG file D) once dropped a whole file from one codec's totals and inflated its AIT lead for weeks. Check that every row of a table sums to the same original size.
7. A same-bytes file that decodes a few percent slower in one pass is noise until proven otherwise. Re-measure it with five interleaved rounds before reporting it: in the v1.6.12-pre check five such files (+2% to +18% cycles, identical instructions) were -0.3% to +0.1% the second time. lzbench's per-file speeds move by up to 6% between passes of the same binary; the exact cycle counts (perf counters, bakeoff/stdh.c) are the number to quote.
8. Document dead ends. COMPETITORS.md records the refuted hypotheses (order-1 gate, window cap, matched literals) so they are not retried blind.

Format policy

The seq block format is a lab format during -pre versions:

• A format break is allowed, but it must be called out in NEWS ("BREAKS THE SEQ BLOCK FORMAT"). A format extension that keeps every old stream valid (v1.6.12-pre's delta lanes) is called out as "FORMAT EXTENSION" and gated with XDIFF_EXT=1.
• The seq level numbers are public API: v1.6.12-pre moved them (1-13 plus --fast), so a number in a doc or a script names a version.
• The HC frame codec stays byte-identical.
• Old LZ6S1 files keep decoding through a decode-only shim, verified against bakeoff/compat_LZ6S1_C.lz6s1.

Windows

./build_windows.sh cross-compiles static x64/x86 binaries. Before claiming something works on Windows, verify on real Windows (Windows 7 SP1 x64 and Windows 10 VMs), not only by cross-compiling.