Skip to content

Component Status

Jump to bottom
YadeWira edited this page Sep 3, 2026 · 36 revisions

Component status

Detailed per-codec native-decode status. See the README for the measured summary table and quick start; see Changelog for the day-by-day history of how each status below was reached.

Known gaps where the bridge is invoked at runtime

(The bridge produces byte-exact output but the C++ does not decode these paths natively.)

Measured state, 2026-09-02 night (after the ring-end copy fix)

Measured on a fresh 155-file real-world corpus (63 MB, 8 categories), because the 61-file corpus is saturated at 488/488 and can no longer detect anything: 1233/1240 byte-exact, zero bridge — and the 7 failures are one file, one cause.

method pass fail method pass fail
-cn 155 0 -cd 155 0
-cf 155 0 -cD 155 0
-cF 155 0 -co 155 0
-cc 155 0 -cO 155 0

The legacy bridge is gone (2026-09-02 night): no code path searches for or runs an original nz any more (audit: strace, zero foreign execve; originals unreadable, 95/95). Every "bridges"/"falls back to the bridge" note below is historical — such a stream is now reported as corrupt.

The image model (decr_param = 3) is PORTED — the corpus is at 1240/1240. NzImageModel (nz_audio.cpp) is FUN_080a9ca0/FUN_080a90c0: NOT a CM but the audio decoder's 2-D sibling on an audio-shaped object — 5 LMS planes (FUN_080bddc0, plane 4 shared across channels, 0..3 per channel), the same bit-count + residual coder (FUN_080a4ea0+FUN_0809baa0, or the bit-count classes + FUN_0809bbf0 when flag bit 0), a 4-stage 4-tap sign-sign cascade fed by the four rows above (FUN_080b6820), an 8-mode pixel predictor over left/above/above-left/above-right (FUN_080b65b0; the encoder only emits mode 2, (l+a+1)>>1) and two history rings. The 51/31/11 header rotation read as "colour planes" during scoping is the raw prefix of a pixel split across chunks (a chunk's misaligned tail bytes are stored raw at its END, the rest of the pixel raw at the start of the next). Per-codec profile from GDB: -cc/-cO flags 0x0f, planes 32/48, bit-count class A; -co 0x07 class B; -cd/-cD 0x02 planes 16; -cf/-cF 0x00 planes 16. Wired into the CM family's mode 3, the -cd 0xf sub-chunk and the -cf bit-3 prefilter-slot block. Byte-exact first build in all 7 codecs; 37 corpus BMPs, 8/16/24/32-bit and PGM/PPM/TGA/TIFF synthetics (the other three detectors reach the same model), and mixed BMP+WAV+text archives all pass. Two unrelated -co/-cO declines surfaced by that sweep (an LZ block after two BWT blocks in a 153 KB 16-bit BMP; a 1.4 MB BMP on the BWT path) were ONE cause, closed the same evening (65a00a5): a BWT block's param15 pass names its match sources as absolute offsets into the accumulated pre-post-filter stream (reference mem->data_org, the LZ window's bytes), not the final output; the port matched into out_data, which param1 had already rewritten almost byte for byte, so the copied bytes were wrong and so was the window the next LZ block read. Found by dumping the original's ring at the failing block's entry (descriptor via the pointer at obj+0x40) and asking where the first differing bytes came from. The 37-BMP sweep is 259/259; no open decode item is known on any corpus or sweep.

The scoping record (superseded where it says "CM" / "planes"): One 24-bit BMP fails every codec except -cn; destroy the BM magic and every codec passes. GDB on the original: the encoder's detector chain (FUN_08089770 and three sibling detectors) puts the block on decr_param = 3 — the value a 357-archive sweep had never seen — and every codec then decodes it with the same function, FUN_080a9ca0 (the CM-family dispatcher's mode == 3 branch for -cc/-co/-cO, the 0xf sub-chunk for -cd, the CM block for -cf), a 64 KB-chunk wrapper around FUN_080a90c0 (765 instructions, header-byte driven). The community reference decodes these blocks as its ordinary CM without reset, which is wrong — and is what this port had copied. Payload: a 4-byte prologue (0x04af = width − 1), the 54-byte BMP header stored raw, then per-plane chunks whose header bytes rotate through three colour planes (the 2.88 MB image gives 1 + 15 + 15 + 16 calls; 960 000 bytes per plane). Most callees are already-ported primitives; the unported core is the hash-bank CM model. One port closes all seven combos, and likely the rare -cd CM sub-chunks on non-image data too.

The -cd/-cD cluster (12 failures earlier the same day) is fully closed — three causes: the prefilter reset skipped on pure-literal LZ chunks; the LZ ring sized round(total/0x10000) where the rule is bytefloat(p1+1)·0x10000 (single-container and per-stream parallel rings); and a match that starts just before the ring end being copied linearly past base+cap into the slack the first wrap zeroed — the original resolves the source once (base + ((pos-off)>>31 & cap) + (pos-off)) in both FUN_08099050 (-cd) and FUN_080982e0 (-cD), while the port copied per byte modulo the ring and read the bytes it had just written at the ring start. Minimal repro 65536+256 bytes (a 255-byte last chunk never starts on the lap boundary). The earlier "two-pass literal model" note was a misreading of FUN_080982e0's second argument, which is the literal stream.

Everything else in this file's older "remaining" notes for -co/-cO is now closed: see the Changelog entry for the two ring-lifetime bugs, which took -cO from failing ~7.5 % of real files to 1 of 155.

  • lzpf stereo prefilter (-cf AND -cF) is now native end-to-end (see the component table below). The remaining -cF bridge case is the non-prefilter LZ77 variant-B path on some inputs.
  • lzhd (-cd/-cD): the coroutine token-LZ is native incl. multi-chunk text, large multi-stream files, and parallel containers (see the component table below). The LZ window is a single per-archive ring (single-container) or a fresh per-stream ring (parallel container) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB; modular helpers so non-power-of-two sizes work) — confirmed by GDB on FUN_08099050 (obj+0x978): 1/3/19/46 × 64 KB for text50/source.cpp/big_code/repeat_3M. The ring is sized to hold the whole compact recon, so its cursor advances monotonically and never wraps for real archives; it persists across the archive's 1 MB output streams. -cD (nz_lzhds) shares this entire front end and adds its own literal model — a per-context MTF rank table + adaptive order-N linear predictor, fed by an MSB-first bit reader + Exp-Golomb integer decoder (nz_lzhds.{h,cpp}) — now ported and native. The decoder still self-verifies against the stored per-file checksum and bridges on any mismatch (defense in depth). Remaining bridge cases: rare CM/BWT sub-chunks (shared by -cd/-cD).
  • optimum1 (-co) single-container decr_param==1 (LZ/CM) blocks are now native (see the component table below); an older theory that large -co/-cO used an unreversed "virtual-stream" DecLZ framing was disproven by GDB tracing against the real binary — the actual blocker was that DecLZ (a port of the community reference decoder) is simply never called by the real binary at all. The parallel multi-stream container (flag 0x0f, >~8 MB) is also native now for BOTH -co and -cO, reusing the same chunk-record format as -cf/-cd's parallel containers. optimum2 (-cO) single-container decr_param==1 blocks are native too now (its richer 8-context literal mixer + LZP secondary predictor ported into nz_optimum2_lz.{h,cpp}). Single-container archives with more than one stream_tag segment ("chain" mode — see the roadmap below) are also native now for both. Still bridging for both -co and -cO: decr_param==0 (BWT) blocks.
  • All five text-transform bits the encoder actually emits are now native: 0x10 (tt16 numbers), 0x08 (word dictionary), 0x04 (HTML closing-tag restoration), 0x02 (insert-LF), 0x20 (escape+RLE) and 0x01 (CR/CRLF). dece (the x86 exe-filter) is now ported too (648df9e, state model corrected in 491a54d), so the entire post-filter chain is native: param2, param1, all six text-transform bits, and dece. Note dece filter state persists across a run of consecutive dece blocks and resets when a non-dece block intervenes — the reference resets per block, which is wrong on ~25% of real dece archives. param1 (AddBytes) is now ported and working for -cc/-co/-cO — a real-world corpus sweep found and fixed a latent word-boundary truncation bug in its BitReader that had silently made it fail on real files whose side-stream length wasn't a multiple of 4 bytes.

Roadmap to 100% native decode

  1. ✅ Done: NZ_NO_BRIDGE=1 flag — when set, FindLegacyBackend* return empty and a missing native path is a hard error (no silent $PATH//usr/bin/nz fallback). This is what native_only_v2.sh uses to measure honestly.
  2. ✅ Done: the -cd cross-chunk / cross-stream LZ window. It is a single per-archive ring of size round(total_output / 0x10000) · 0x10000 (min 64 KB; GDB-confirmed on FUN_08099050 obj+0x978) that persists across the archive's 1 MB output streams (NzCdDecodeStream threads the ring position and the file-absolute output offset). The ring is sized to hold the whole compact recon, so it never wraps for real archives. This made multi-chunk text and large multi-stream files (a 7-chunk source file; 2–3.5 MB code/text/base64 incl. heavily-repetitive input) byte-exact native, with a checksum self-verify → bridge fallback as defense in depth.
  3. ✅ Done (-co/-cO, single-container AND parallel-container): ported the real linux32 -co/-cO LZ/CM cores (nz_optimum_lz.{h,cpp}, nz_optimum2_lz.{h,cpp}), LZMA-shaped LZ77 engines with 4 repeat-offset slots — not DecLZ, which is never called by the real binary. Also ✅ done: the tt_flags & 0x02 (InsertLF) text-transform bit for both -co and -cc; and param1 (AddBytesFilter) for -cc/-co/-cO. Remaining optimum work:
    • ✅ Done: decr_param==0 (BWT) for both -co and -cO, in both shapes. The block header layout was missing entirely (a non-CM decr_param==0 block carries param7 when param6 is set, a u32 inverse-BWT start position, and params 14/15 — none of which exist in the LZ layout). With param6==0 there is no entropy layer at all: the encoder found the BWT output incompressible and stored it raw, so the block expands to exactly its payload size and only the inverse BWT (NzBwtUntransform) runs. With param6==1 the output is wrapped in 256 independent per-leading-symbol buckets, each an arithmetic-coded move-to-front rank stream plus an optional byte-wise RLE expansion (NzBwtDecodeInput). New files nz_bwt.{h,cpp}. native_only_v2 -co/-cO 7/10 → 9/10 each, TOTAL 71 → 75/80; real-world 52-file corpus -co 29→32, -cO 27→31.
    • A compiler-visible latent bug inherited from the reference, surfaced by the above. The reference's rank-insertion loops read one element past their own uint8 P[256]: for (; new_c >= C[P[k + 8]] && k != 249; k += 8, new_c += 8). At k == 249 the left operand reads P[257], and the value cannot change control flow (the k check ends the loop that same iteration) — so it looks harmless, and is at -O0/-O1. At -O2 gcc is entitled to assume P[k+8] is in bounds, hence k <= 247, hence k != 249 is always true, and to delete the bound. The loop then runs away (k observed reaching 2313) and P[k] = last_rle writes far past P[255], corrupting the C[] array the compiler had placed immediately after P — surfacing much later as an absurd num_rle and a declined block. Testing the bound before the read removes the UB and restores the intended semantics. Found with ASAN (which flagged only the read, since at -O1 it does not exploit the UB) plus an A/B revert: with the reference form a real 1.5 MB BWT block fails, with the bound-first form it decodes byte-exact. The reference decoder is presumably miscompiled the same way at -O2, which may be part of why it is known to get some -co/-cO edges wrong.
    • ✅ Done: BWT param14/param15 follow-on transforms (NzBwtParam14/NzBwtParam15, ported from DecodeLZ_Param14 at NZ_LZ.cpp:543 and DecodeParam15 at NZ.cpp:843). Both run after the inverse BWT and before the shared post-filter chain, and both are LZ77 passes that find matches by scanning the byte stream for a two-byte escape tag instead of coding literal/match flags: param14 uses 0xfe 0xf1 + a selector byte with offsets relative to the output position and four repeat-offset slots; param15 uses 0xfe 0xf0 and names its source as an absolute offset (four raw big-endian one's-complement bytes taken from the byte stream) into the whole accumulated output stream, so a match can reach back into earlier blocks. param14 turned out to be common, not an edge case — roughly 20 of 220 sampled real files carry one, across audio/music/image formats. Note this param14 is NOT nz_cd_tokens.cpp's NzCdParam14 (the -cd char-class space-insertion transform) — same name, different algorithm.
    • ✅ Done, and the more interesting half: non-LZ block output must be fed into the LZ window. The first real file to get past the new param14 gate still failed — on a later, ordinary LZ block. In the original the LZ window is the shared accumulated-block buffer that every block writing into it advances (mem->data += size), which includes a BWT block's post-param14/15 output; only audio blocks are exempt, because DecodeFromStream returns before touching the window for those. This port keeps a private ring that only DecodeBlock ever wrote, so a later LZ match reaching back into a BWT block's output read stale ring bytes. Fixed with FeedWindow on both optimum decoders, called with the block's pre-post-filter bytes (which is what the window carries). Same class of bug as the CM stored-block desync fixed in 25d2f75. Shipped 296ee88. Measured on one identical 60-file real corpus with a baseline binary built at caeb56d: -co 43→52, -cO 45→54 (+9 each, 72%→88%) — the largest single jump of any fix in this project.
    • ✅ Done: single-container "chain" mode (multiple stream_tag segments in one archive, directly concatenated with no separator). A real 60 KB .doc file surfaced this — its first segment covered only 10229 of 60416 declared output bytes, and the very next byte was a second, independently valid stream_tag varint. An earlier session's negative "chain mode doesn't exist for -co" conclusion was based on insufficient fixtures (large synthetic files only, never a small real multi-block document). Fixed by looping over segments with one persistent decoder shared across the whole entry. Real-corpus sweep: -co 20→26/56, -cO 21→27/56.
  4. Port the remaining rare post-filter (dece exe-filter) and the CM stereo-audio variant.
  5. Partly done — and it turned out to be two unrelated gaps, not one. NanoZip has two distinct audio bitstream families:
    • decr_param==2 blocks (-cc/-co/-cO) — decoded by the reference's AudioPred, now ported as nz_audio.{h,cpp}. These blocks also have their own truncated header shape (a mode2_type byte, param6 forced to 1, size18, then STOP — no staged-checksum count and none of the param2/param1/param16/tt/dece fields), which is why every audio-bearing archive previously died before its first block trace: the ordinary parser read mode2_type as param6 and walked into the next record. decr_param==3 shares that shape minus mode2_type; it is the IMAGE model (see the 2026-09-02 note at the top), ported as NzImageModel. ✅ -cO is byte-exact on stereo and mono at 8/16/24-bit and on a 6 MB multi-block file exercising mode2_type=1 plus cross-block predictor state; native_only_v2 -cO 9/10 → 10/10, TOTAL 75 → 76/80. ❌ -co and -cc still decline — their output is bit-identical to the reference decoder's on the same payload (verified by building the reference AudioPred standalone), so the port is faithful and the reference itself diverges from the real binary on the shapes those two emit. The discriminator is which linear predictors the encoder enables: -cO's working set is {0,1,3}, -co adds 4, -cc adds 2 and 5. Sweeping the third pair's order over 13 values does not fix it → needs GDB against linux32/nz.
    • -cd/-cD audio — a completely different family: it surfaces as a chunk-header varint 0x0c, i.e. the same construct as the already-ported -cf/-cF audio block (0x04) but with bit 3 set, which selects real-binary FUN_080a9ca0 instead of the ported FUN_080a5bb0. AudioPred is not the right decoder here (its 2-byte-length-prefixed bit-count framing does not match). Unported.
  6. Extend real stereo/multichannel audio decode to -cd/-cD — a real-world corpus sweep found these codecs have near-zero coverage on real audio content (unlike -cf/-cF, whose stereo prefilter was fixed to 9/9 on the same corpus). These codecs don't have a dedicated prefilter like -cf/-cF — the gap is presumably in their own entropy/CM paths or the shared decr_param==0/BWT path.
  7. ✅ Done: integrty.doc declining under -cc was initially misattributed to param2's NzBwtRleDecodeU32 RLE expander (the function that returned false), but tracing one level upstream showed param2 was correct — the CM chunk feeding it had already produced garbage output partway through. Root cause: when a decr_param==0 chain has a param6==0 STORED (raw) chunk sitting between two CM-coded chunks, the CM model's rolling byte-window/hash context must still observe those stored bytes to stay in sync with the true output position, even though they were never CM-arithmetic-coded — the port never did this. Confirmed by building the community reference decoder standalone and reproducing the identical corruption on the real archive's exact bytes, then confirming the fix (feed stored bytes through a model-update-only path, NzCmFeedByte) against that same reproduction. Real-corpus sweep: -cc 31→32/56.

Component table

Component Cloned? Notes
CLI structure (l/t/x/a/s, all switches) ✅ 100% Full parser
Archive format (header, entry table, stream families 0x2b/0x3b/0x4b) ✅ 100%
Store (-cn) ✅ 100% Single-block stores are a contiguous tail copy. Files large enough to split (> ~0x30000) are stored as multiple blocks — [varint (len<<4)|0][raw] with a per-block checksum trailer between blocks — reassembled by TryAssembleStoredBlocks (-cn 9/10→10/10; the prior miss was a 215 KB source file stored as 196608 + 18566).
lzpf decode LZ77+arith path (-cf/-cF) ✅ native (10/10 measured) LZ77+arith (variants A 13-bit hash, B 24-bit hash) + literal + raw-bytecode block modes native. Native byte-exact on most real inputs under NZ_NO_BRIDGE=1. Large single-container multi-stream -cf/-cF (1–8 MB) decode byte-exact: the output splits into ~1 MB compressor streams chained [stream_tag][data] with an inter-stream whole-output checksum record (tag 0x45/0x47/0x26); dict/hash state persists across all streams. The parallel multi-stream container (header flag 0x0f, >~8 MB, multi-threaded encoder) also decodes byte-exact: N independent nz_lzpf streams, each a group of chunk records (type-1 slice size, type-10 u32 output offset, type-5/7/6 slice checksum, one or more type-0 compressed chunks concatenated into the stream), each decoded and placed at its offset and required to tile the whole output, every byte slice-checksum-verified. The sliding-window dict capacity is a multiple of 64 KiB the encoder picks by its threading (one of (p1+1)·64 KiB, or total rounded down/up to 64 KiB / 128 KiB; GDB-confirmed on FUN_080b6bb0 obj+4), so the decoder tries each candidate and adopts the first whose decode matches the slice/whole-output checksum (an unmodelled window wrap is rejected, never emitted). Mixed-binary content (real-world tars with both text and incompressible/random data) also decodes byte-exact: literal blocks now backfill hash_table after every literal run (legacy FUN_080b6d90/FUN_080b6cf0 — sparse, every 101 bytes, or dense, every byte, selected by a bit in the block header; variant B additionally zeroes its byte_buffer_8k side-table per FUN_080b6c20). Without this the table's untouched init value (3) leaked into later f6/f8/medium-match copies whenever a match referenced a hash bucket only ever touched during a literal run — byte-correct through many blocks by chance, then producing sparse single-byte corruption the moment a genuinely-collided bucket got referenced. Root-caused by GDB-diffing the decoder's hash_table against ground truth at the exact block boundary where output first diverged. The "8/8 100%" figure is from the low-entropy coverage_matrix.sh fixture only and does NOT reflect real data — see the measured table in the README. Variant B hash init=3 fix (commit 049d041) closed the multi-file silent corruption.
lzpf prefilter+arith path (-cf) ✅ mono + stereo native FUN_080a5330 + LPC filter (FUN_08095d90) ported; mono AND stereo audio byte-exact. The stereo path was closed end-to-end (stereo_lms_cf.nz → stereo_lms.wav, both blocks byte-exact, -cf native_only_v2 9/10→10/10). The full FUN_080a5330 stereo flow is now reproduced in DecodePFBlock: residuals are PLANAR (ch1=[0,per_chan), ch2=[per_chan,2·per_chan)) decoded by two per-channel DecodeArithBuffer calls (each reads its own Huffman header — a single n_elems call mis-locates the side stream by ~1.4 KB); the predictor-init reads a leading bit G (iStack_50078, gates the inter-channel LMS) then per channel [active(1), order(3 if active)] (order = bits+8); per-channel LPC with two persistent predictor states (pred/pred2, threaded across blocks); the verified inter-channel LMS FUN_08096e20 (ApplyLmsInterChannel, regression test TestLmsInterChannel); and FUN_080a50c0 reconstruction (ReconstructStereoSamples: per-channel delta-integrate + L/R (channels==1) or mid/side (channels==2) interleave). -cF (lzpf B / nz_lzpf_large) stereo is also native: it uses the SAME FUN_080a4ea0 arith as -cf (not a vtable path), differing only in the LPC filter order — FUN_08095d90 obj+0x1c08 is 8-tap for variant B vs 4-tap for variant A (GDB-confirmed), so LpcPredictor is now N-tap (8-tap adapts every sample, 4-tap every other). -cF native_only_v2 8/10→9/10. A real file-format-sample sweep later found the single-fixture validation above had missed real bit-depth/channel variation (5/9 on a 9-file real stereo/multichannel WAV corpus spanning 2/6 channels, 8/16/24-bit): fixed 4 real bugs — tail-remainder bytes silently dropped instead of copied verbatim; the generic (non-int16-fast-path) reconstruction always did plain L/R interleave instead of mid/side for channels==2; the LMS per-object adaptation-shift bits were never read/applied; and LPC/LMS predictor state wasn't reset across a literal/LZ77 block sitting between prefilter blocks (GDB-confirmed on the real binary: state is exactly zero after every non-prefilter block, carried forward after every prefilter block, 16-for-16). Real-corpus audio sweep now 9/9 for both -cf and -cF.
lzhd decoder (-cd/-cD) ✅ LZ (both -cd raw-literal and -cD MTF+predictor literal models) + block-RLE + raw-store + pure-literal + exe + text-pipeline (param14/line-RLE/CRLF/word-dict) + multi-chunk text + large multi-stream files + parallel containers native; only rare CM/BWT sub-chunks bridge The real linux32 -cd is a coroutine token-LZ (NOT the reference DecLZ). Fully ported in nz_cd_tokens.{h,cpp} and validated byte-exact against the binary: token assembler (FUN_080aa070), reconstruction with trailing-literal flush (FUN_08099050), per-column RLE run-expander (FUN_080acb90, thr=1 for the LEN column / 0 for LIT·OFF), param14 text transform (FUN_080a0ff0), the bounded-varint header (FUN_080b1dc0), and the integrated chunk/stream/block decoders (NzCdDecodeLzChunk/NzCdDecodeStream/NzCdDecodeBlock). Columns/literals reuse DecodeArithBuffer. The reconstruction runs over a single per-archive ring (FUN_08099050, obj+0x978) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB) — GDB-confirmed across text50/source.cpp/big_code/repeat_3M = 1/3/19/46 × 64 KB. The size need not be a power of two, so the ring uses modular helpers (not a bitmask). The ring is sized to hold the whole compact recon, so its cursor (obj+0x980) advances monotonically and never wraps for real archives (the reset/wrap path is a safety fallback). Large files split their output into 1 MB streams; the ring is allocated once and persists across streams (NzCdDecodeStream threads ring_pos and the file-absolute output offset), so cross-stream matches resolve correctly. Wired into the extract dispatcher and BYTE-EXACT end-to-end under NZ_NO_BRIDGE=1 for: token-LZ (recon == file), block-RLE (flag &2, post-recon run-length re-expansion of collapsed zero-runs), raw-store (per-column b0&1==0 and flag-&1-clear literals = verbatim bytes), pure-literal (no LZ tokens — the whole window is one literal stream; generator picks this when size_field==0 or v2==0), exe (flag &4: a BCJ-style x86 E8/E9 call/jmp address un-transform, NzCdExeUnfilter), and the text pipeline (flag &8, FUN_080a3c90): param14 (NzCdParam14), line-RLE (FUN_080a2f20), CRLF EOL (FUN_080a19b0), and the word-dictionary transform (FUN_080a0a00 = the reference TransformText_1_Dictionary + dict/char-trait tables, in nz_cd_texttransform_dict.cpp). Verified on text (map.txt), binary (image.cat), an ELF (elf.bin, flags=3 block-RLE), an EXE (play.exe, flags=5), atoll (multi-chunk &8), word-dictionary text, multi-chunk text (text_50k/repeat_100k/CRLF), a 7-chunk source file (single-stream 192 KB ring), large multi-stream files (2–3.5 MB code/text/base64, incl. heavily-repetitive input), and a varied real corpus spanning images/audio/video/executables. As defense in depth TryDecodeLegacyLzhd self-verifies the decoded output against the archive's stored per-file checksum and returns false on mismatch, so any unforeseen edge falls through to a byte-exact bridge decode (no silent corruption, and NZ_NO_BRIDGE native-only is a provable correctness signal). The parallel multi-stream container (header flag 0x0f, >~8 MB, mirroring the -cf/-cF parallel format) also decodes byte-exact: N independent nz_cd streams, each with its own FRESH ring (unlike the single-container case, where one ring is shared across the whole archive — confirmed empirically: each parallel-encoder thread owns its own nz_cd instance), sized per-slice via the same round(slice_size/0x10000)·0x10000 formula; each stream's type-0 chunks are already-delimited raw DecLZ blocks (no concatenation needed, unlike lzpf), decoded one at a time into the slice and verified against its own per-slice checksum before being tiled into the assembled output. Verified byte-exact on parallel containers up to 20 MB (text, random, and mixed text+random content). -cD (nz_lzhds = "lzhd strong") shares this ENTIRE front end (same token assembler, same column decoding, same LZ match/rep-cache model — even the same FUN_08099050-equivalent recon loop, dispatched via a vtable slot) and differs only in the literal coder: a per-context MTF rank table (256 contexts × 64 bytes: 32-slot rank table + 32-byte presence bitmap, seeded to the identity permutation with bitmap bits 0–31 set, not all-zero) plus a 4-stage adaptive linear predictor (pred = (w3·h3+w2·h2+w0·h0+0x200+w1·h1)>>10, sign-updated toward the residual), fed by an MSB-first bit-reader + two-level Exp-Golomb integer decoder over a dedicated -cD-only bitstream (one raw length byte + that many raw bytes, sitting between the token assembler's extra-bits blob and the literal column in the chunk layout — GDB-confirmed, not documented anywhere in the archive format itself). Ported in nz_lzhds.{h,cpp} and wired into both the single-container and parallel-container -cD paths in TryDecodeLegacyLzhd. Remaining bridge cases: rare CM/BWT sub-chunks (shared with -cd, e.g. stereo audio). The old DecLZ (nz_lzhd.cpp, nzdec_v0) is the wrong format for linux32 -cd/-cD.
optimum decoder (-co/-cO) ✅ both native single-container AND parallel-container LZ / ⚠️ BWT still bridges Three RE sessions of GDB tracing against the real linux32/nz binary found the real -co/-cO decode core is not DecLZ (the community-reference codec that name was borrowed from is never called) — it is a distinct LZMA-shaped LZ77 engine with 4 repeat-offset slots (rep0–3), a unary/Elias-gamma-style length/distance code, and an adaptive literal mixer sharing its low-level binary range coder with nz_cm.cpp. -co (FUN_0809e600, the "compact" nz_optimum1 variant, 4-context literal mixer) ported in nz_optimum_lz.{h,cpp}; a real port bug (the literal mixer's context-C seed sign-extended a byte the real binary zero-extends before shifting) was found and fixed via GDB divergence-tracing against a real failing archive. Also ports NzTextTransformRle (tt_flags & 0x20) and NzTextTransformInsertLf (tt_flags & 0x02). Native for -co single-container AND parallel-container (flag 0x0f, >~8 MB — reuses -cf/-cd's already-solved chunk-record format verbatim) archives with decr_param==1 (LZ/CM) blocks; still bridges on decr_param==0 (BWT). -cO (FUN_080a5d90, the "large" nz_optimum2 variant) shares the same backbone but a materially richer 8-context literal mixer plus a rolling 3-byte-hash LZP-style secondary predictor with no -co analog — ported in nz_optimum2_lz.{h,cpp}, reusing nz_cm.cpp's own kLzModelLNext/CM_Input_Bit primitives and -co's own dumped tables directly (GDB-confirmed byte-identical, not re-embedded) rather than re-deriving them. Wired into TryDecodeLegacyOptimum for method_p0==6, checksum-gated identically to -co. Found and fixed 3 real port bugs via GDB ground-truth (a units-to-bytes conversion in the dispatch-bit's second APM stage, a missing counter-based addressing scheme in rep-slot selection, and a copy-paste wrong base address in the length decoder's extra-bits loop) plus a 4th, subtler divergence that only manifested once mixer/weight state moved past the cold-start case (needed a longer, real-world-shaped stress fixture to even surface). Native for -cO single-container AND parallel-container (flag 0x0f) archives with decr_param==1 blocks — the parallel-container branch is shared verbatim with -co (the decoder type is selected once via a small closure since it's fixed per-archive, not per-stream), no new RE needed. decr_param==0 (BWT) still bridges for both -co and -cO.
cm decoder (-cc) ✅ native byte-exact Native CM decoder (NZ_CM.cpp, 1100 LOC). The documented "byte-26" divergence is FIXED (2026-06-08): it was a one-line port error in CM_Input_Bit — factors0_err used truncating factors[0] / 16 instead of the reference arithmetic shift factors[0] >> 4 (differs for negative values, flipping the factor[7] zeroing condition). Found by a per-bit next_probability diff vs a compiled reference oracle. With the full post-filter pipeline ported — param2 RLE (NzBwtRleDecodeU32), param1 AddBytes delta filter (NzAddBytesFilter), tt08 dictionary, tt16 number-transform (NzTextTransformNumber) — plus multi-chunk decoding (CM state persists across consecutive type-0 chunks), stored blocks (param6==0 = raw payload, for incompressible data), and NzTextTransformInsertLf (tt_flags & 0x02, shared with -co), -cc decodes byte-exact natively (no bridge) on source, numbers, dates, IPs, prose, markdown, random/large multi-chunk: 13/13 on a comprehensive sweep, 9/10 on native_only_v2 (the one miss is the deferred stereo audio-CM variant). NzAddBytesFilter's BitReader had a latent word-boundary truncation bug (rounded its side-stream length down to a multiple of 4 bytes instead of using the exact byte count, matching the reference exactly except for this one truncation) that a real-world corpus sweep found — it silently failed (clean decline, never wrong bytes) on any real file whose param1 side-stream length wasn't a multiple of 4, undetected by the synthetic test corpus; fixed, and the fix transparently benefits -co/-cO too since they call the same function. Cross-chunk continuation across a stored block was also missing: when a param6==0 stored (raw) chunk sits between two decr_param==0 CM chunks, the CM model must observe those stored bytes (model-update only, via NzCmFeedByte) to stay in sync with the true output position for the next CM chunk — found on a real file (integrty.doc), fixed, -cc real-corpus 31→32/56.
Encode for all methods ⚠️ functional Native BWT/store/literal writers; the legacy compression bridge was disabled in commit 049d041 (IsInternalLegacyCompressionBridgeCompressor returns false unconditionally). Codecs the native encoder cannot handle now produce an explicit error. All 8 methods byte-exact round-trip via the native encoder for low-entropy inputs.

Fixture-based benchmark (coverage_matrix.sh)

The coverage_matrix.sh test uses a deterministic AES-CTR-of-zeros fixture (low entropy). On that fixture:

native_strict_percent = 100% (8/8) — no bridge, no compat, no original binary subprocess.

Low-entropy input rarely triggers the prefilter+arith block mode, so the fixture passes fully native even though that path is not yet ported. Real-world compressible data (text, source code, binaries) will hit that path and fall back to the extract bridge for -cf/-cF.

Important caveat: native_strict_percent measures "no [bridge] / [compat] log line in stdout". It does NOT measure "the C++ code path actually ran the native decode vs silently called the legacy binary via FindLegacyBackend". A fixture that triggers the prefilter+arith block mode shows the bridge being invoked at runtime but still reports native_strict_percent=100 because the bridge produces byte-exact output. To verify the code path is truly native, use tests/native_only_v2.sh (NZ_NO_BRIDGE=1) — see the README's measured table, which is the honest number.

Real-corpus standing (60 files, 2026-09-01, all measured on the same corpus)

codec pass codec pass
-cn 60/60 -cd 60/60
-cf 60/60 -cD 60/60
-cF 60/60 -cc 60/60
-co 60/60 -cO 59/60

479/480 overall. Seven of the eight codecs are perfect on this corpus; one file remains. All five codecs that are perfect here — -cn, -cf, -cF, -cd, -cD — cover the whole 0x2b family plus store. Every remaining failure is in -co/-cO/-cc.

(2026-09-01: -cD stage-index fix c831c31, -cf/-cF exe-filter fix f885e1c, prefilter hash-backfill fix eb66e8c, prefilter sub-chunk cd5e3f3, and the two cross-chunk state fixes 7a4d4c5 that took -cd 58 → 60 and -cD 52 → 60.)

Measuring every codec on ONE corpus is what surfaced the -cD deficit (then 34/60); comparing numbers across differently-generated corpora had been hiding it. Four files (Moly, summer.php, M05.AMF, RESOURCE.001) fail across most codecs and are the shared hard set.

Decode: what is left (audited and closed 2026-09-01)

Four gaps were open at the last audit. All four are closed, and all four turned out to be the same shape of bug: a field or branch combination the common path never emits. None was visible to a test matrix that only ever built archives the default options produce.

Closed: BWT stored bucket (0b8a669)

-co -- the DEFAULT compressor -- and -cO declined on any file mixing high-entropy and compressible data from about 1 MB up: an installer, a disk image, a PDF with embedded JPEGs.

A BWT block with its entropy layer (param6 == 1) splits into 256 buckets by leading symbol, and the bucket table is read backwards as (out, in) varint pairs. When the encoder cannot compress a bucket it stores it verbatim and writes in == 0 to say so -- the bucket's input length IS its output length and there is no per-bucket header. Reading that as a literal zero left the bucket's bytes unaccounted for, so every later bucket was distributed from the wrong offset and the first one that did carry a header parsed its neighbour's data.

The community reference has the same gap, so reading it could not settle this. What settled it was tracing the raw table reads: the entries with in == 0 consumed their varint bytes like any other, so the zero was written, not misread -- and assuming in == out for them made in_pos land within two bytes of the payload end. No GDB needed.

Closed: lzpf dictionary capacity is derived, not searched (15005b4)

-cf/-cF archives written with -hn or -nm declined outright. The capacity was found by SEARCHING five guesses -- ceil/floor of the total over 64 KiB and over 128 KiB, plus (p1+1)*64 KiB -- and keeping the first whose decode matched the stored checksum. That only worked because a wrong capacity is harmless until the window wraps, and it could not work at all with no checksum to adjudicate.

It is bytefloat(p1+1) << 16: m = (p1+1) & 0xf, s = (p1+1) >> 4, and if s then m = (m+16) << (s-1). The same mantissa/exponent byte the -cc window size and the optimum window size already used. Linear in p1 for small values and exponential above, which is why a plain (p1+1)*64 KiB fits every small archive and misses a large one (p1=38 means 46 × 64 KiB, not 39). Found by tabulating (total, p1, winning cap) across the real corpus: every row matched, including the four where ceil-of-total was wrong. Verified as the ONLY candidate against all three suites before being made the primary path.

Closed: a column's codec bit and its RLE size-field are independent (9809520)

-cdP and -cDP decoded a multi-block archive to wrong bytes (caught by the lzhd self-verify, so it declined rather than writing). Each token column starts with a selector byte whose bit 0 picks the codec (arith vs raw-store) and whose remaining bits are a size-field for a per-column RLE run-bit region. These were treated as mutually exclusive, so a raw column WITH RLE (b0 == 0x02) consumed the wrong byte count and everything after it parsed from the wrong offset. -cd/-cD never emit that combination -- all their columns are b0 == 0x00 -- which is why eight selectors' worth of testing never saw it.

The tell was arithmetic: a correct chunk has litsum + summlen == out_size exactly.

Same commit closed a real memory-safety bug in the same place: the chunk cursor could advance past the end of its stream, because DecodeArithBuffer reports how many bytes it LOGICALLY consumed and that can exceed the buffer it was handed. Once cur passed end, (size_t)(end - cur) underflowed to ~2^64 and the next call was handed a two-exabyte "size" -- an out-of-bounds read reached by nothing more exotic than an archive written by a different compressor variant.

Closed: a parallel container holding several files (a441f48)

nz a -r <folder> produces this for anything over about 8 MB, so it is an ordinary shape.

A stream is a WORKER, not a file. It can carry slices of two different files; one file's slices are spread over several streams; and the type-10 offset is relative to the FILE, not the output -- in a two-file archive both files have a slice at offset 0, and a missing type-10 record means 0 (the encoder omits it for a slice that is the whole file). That is why the single-file tiling paths could only be gated off, never reused: tiling a multi-file archive would have overlapped the files, and the per-entry check cannot catch it because a split file has no whole-file checksum, only a per-slice one (which is why the original lists such a file as n/a).

Within a stream, each type-1 table names one file and length, the following type-10 gives that slice's offset inside its file, and the stream's checksum record carries one value per slice in the same order. A stream decodes as ONE unit -- its data records are a single chain whose output is its slices concatenated -- and each slice lands at its file's base plus its own offset. One generic assembler plus four small per-codec closures.

The file ORDER is not record order either: the first table in the file belongs to whichever worker started first. It is the order of the stream whose tables name the most distinct files -- the worker that saw the whole job.

The one failure left

-cO on doc/Fonts Poster-color.dp, 1 of 480 corpus cases.

It is our bug, not NanoZip's. Verified 2026-09-02: the original round-trips that archive losslessly (nz a -cO then nz x, cmp clean). Nothing had checked this, because the corpus sweep compares our decode against the original's and would never notice the original losing data. -co decodes the same content fine, so it is specific to optimum2's own literal model (the 8-context mixer plus LZP secondary predictor).

Chunk 4 of block 2 is correct for 2923 bytes and then emits 0x80 where 0x40 belongs at output offset 122018. A literal is coded MSB-first, so the very first bit of that literal decodes wrong -- one bad probability at one position, not accumulated drift. The visible FAIL@distance pos=10568 acc=385363 capacity=131072 is ~7.6k of downstream damage later.

Repro bisected from 160000 to 132656 bytes of input (54794-byte archive), saved with the full analysis in ~/.cache/nzre_tools/optimum2_dp/. Localisation takes about a minute now: NZO2_DUMP_PARTIAL writes a failing chunk's partial output (a failing chunk copies nothing out, so it used to be invisible) and NZOPT_DUMP_FAILBLOCK the whole failing block; the pre-checksum dump also used to sit after the size gate, so a part-way failure dumped nothing at all.

Refuted, do not redo: EnsureHeadroom (five variants). Inherited block state (the 580-byte model diff at block-2 entry is almost all pointer fields from a captured cold blob). "Window size is ruled out" -- that rested on an INVALID experiment: a window override changes the ring-position trajectory and hence the LZP table contents, so a big-window run diverges somewhere else entirely for unrelated reasons. And "it fails once the ring evicts": inputs of 130500/131000/131072/131500/ 132000/132500 bytes all pass against a 131072 ring while 132656 fails, so the trigger is content- and length-dependent (a longer input changes the encoder's chunk layout), not the first eviction.

The blocker is unchanged: comparing our model state at the divergence against the original's needs GDB on FUN_080a5d90's chunk loop. NZOPT2_TRACE_SETUP already prints our side per literal and nothing in it looks anomalous on its own, which is exactly why the original's values are needed.

Still absent on purpose

  • w32c (self-extracting archive creation).
  • -x, -br, -bw, -t<n> and -m are parsed but inert (single-threaded, unbudgeted).
  • a writes this project's own container, which the original rejects as "incompatible version (0.90)", and does not compress. Only decode is implemented.

Latent / never observed

  • 0xd / 0xe chunk nibbles: reachable, never seen in any sweep.
  • decr_param == 3: 0 occurrences in 654 blocks across 357 archives.
  • tt bit 0x20 for -cc: wired, no corpus archive sets it.
  • The -cD stored-chunk model reset is scoped by measured behaviour, not decompile evidence.
  • FUN_0809bdc0's initial table is a rule DERIVED from the state its builder leaves.
  • An archive of ONLY zero-byte files extracts to nothing in the original while this decoder creates the empty files (an empty member beside a real one matches). Cosmetic, not a defect.

Measured the way a user runs it, not just the way it is convenient to measure

Every suite sets NZ_NO_BRIDGE=1, which is the only honest way to measure native decode. It is also not how anybody who downloads a binary runs it, and that gap hid a bug that inverted the whole point of the project: the CM path cross-checks its native output against a bridge decode as a safety net, and it ran that check whenever the bridge was enabled by configuration. When the check could not run -- because no legacy binary is reachable, which is the normal case for a user -- it left the result "unverified" and declined. Every -cc archive failed by default, while the decoder was perfectly capable of decoding it and every suite reported green.

"Enabled" and "available" are different things. The native result is already gated on the stored per-file checksum, so the comparison is a bonus; when it cannot run, the native decode is trusted.

Found by running the published binary under env -i with no variables set, from a directory with no legacy nz anywhere near it: 27 of 29 archives produced output, and both missing ones were -cc. multifile_v2.sh now has a phase that does exactly that and byte-compares against the oracle. Lesson: a test harness that configures away the default path is not testing the product.

Robustness against input that is not a valid archive

Fuzzed 2026-09-02 before the v0.3.0-pre binaries went out (7e05735): 761 cases under AddressSanitizer + UBSan -- truncations at eight fractions of every seed, single-bit flips weighted toward the header and metadata run, 48-byte corruption runs, plus non-archives renamed .nz (an ELF, a zip header, random bytes, an empty file, a bare header). Four real problems, all reachable from a file someone could hand you:

  • An out-of-bounds WRITE. NzTextTransformNumber took an out_cap and ignored it outright. Driven by both a byte stream and an arithmetic side-stream, a corrupt pair emits far more than the caller declared: a 191-byte mutated archive walked off the buffer and segfaulted.
  • A 214-second denial of service on that same 191-byte file, before the crash. The invariant that fixes this class: bound the work against the DECLARED OUTPUT, per ENTRY, not per call. A valid CM decode needs exactly 8 bit decodes per output byte no matter how the chunks are cut, and a per-call bound still lets a corrupt header multiply the work by inventing chunks. 214s -> 1s.
  • A corrupt Huffman table indexing one past symbols[256].
  • Two signed-overflow sites in the prefilter cumulative sums -- benign on x86 today, but -O2 has already deleted a loop bound in this project over signed-overflow UB.

761/761 clean afterwards. Worst corrupt case 3.7 s natively; a non-archive is refused in ~10 ms.

Two traps worth remembering. A 25-second per-case timeout mislabelled the worst bug as a "hang" -- it was a 214-second run ending in SIGSEGV, so always re-run an apparent hang with a long timeout before believing the label. And a backtrace's innermost frame can be a leaf with no loops at all (CM_Input_Bit here); the spinning loop was two frames up, so sample the process several times rather than trusting one bt.

Test coverage

tests/multifile_v2.sh is what closed the blind spot: all twelve compressor selectors the binary's own usage lists (not the eight this project measured for most of its life), nine archive shapes compared as whole extracted TREES -- contents, mode AND mtime -- plus extraction under the metadata switches and the listings. 108/108 extract · 36/36 switches · 72/72 listings. native_only_v2.sh 96/96, real corpus 479/480, units 2553 + 4 + 2.

The method lesson. Every one of the four gaps was found by widening a DIMENSION of the test matrix, never by staring at code: twelve selectors instead of eight, MB-scale fixtures instead of KB, the option switches, parallel containers, and edge shapes (empty files, repeated paths, UTF-8 names). Read the tool's own usage before believing the matrix is complete.

Open work (as of 2026-09-01, after 85d046e)

Synthetic suite: 88/88 — 100%. All eight codecs at 11/11, zero bridge, no fixture needs the legacy binary for any method. Real corpus (60 files, all codecs on the same corpus): -cn 60 · -cf 60 · -cF 60 · -cd 60 · -cD 60 · -co 60 · -cc 60 · -cO 59 — 479/480.

One failure is left in the whole project. There is no unported function, block kind, chunk kind or post-filter remaining.

1. -cO on doc/Fonts Poster-color.dp — one mis-decoded literal bit

The only failure left in the project. Localised twice over, so this is a warm start.

Where it goes wrong (output side). Block 1 (19219 bytes) is byte-exact. Block 2 is 122944 bytes as four 0x8000 chunks; chunks 1-3 (ring 19219..117522, 98304 bytes) are completely correct — 91439 literals, zero wrong. After the ring wraps, chunk 4 runs 2922 more bytes and 59 more matches correctly, then emits one wrong literal byte at chunk-4 ring position 2923 (file offset 122018): 0x80 where 0x40 belongs. Everything after is downstream damage; the visible error is FAIL@distance pos=10568 acc=385363 capacity=131072 about 7.6k later (a bogus slot 18, where a 2^17 window admits at most slot 16).

Where it goes wrong (state side), measured 2026-09-01 (11f94a6, corrected in 54a0f05). The real object is arg0 of FUN_080a5d90; it is stable across runs, and a 48-byte window at +0x90 matches our mem_ dump byte-for-byte, which establishes that the two layouts line up.

At block-2 entry the raw diff is 580 of 17 313 920 bytes — but almost all of it is not model state:

  • Pointer fields. The captured cold blob has heap pointers (0xf4../0xf5..) and static ones (0x08..) baked in from whatever run it came from; this port never dereferences them. The value pairs that looked like drifted probabilities — 63287 vs 62804 — are the high halves of two different heap addresses.
  • A per-slot statistic. The record base is 0x103c140 + (slot - 4) * 0xc0 (confirmed at 0x080a8306, which zeroes it; for slot 5 that is exactly 0x103c200). Offset 0 of each record is a u32 the real code zeroes before its bit loop and increments once per bit, read only to increment (watched at 0x080a8311 / 0x080a8430 / 0x080a8449). The real values 1,2,3,4,5,6,7,8 across consecutive slots are per-slot bit counts.

Filter those out and the cold states differ in two u32 counters (0x103f3c0: 17 vs 18, 0x103f3c6: 1 vs 2 — neither referenced anywhere in nz_optimum2_lz.cpp) plus the 64 cells of the port-invented align table.

So the model state at block-2 entry effectively MATCHES, and the earlier "latent inherited state" framing was wrong. The divergence is introduced inside block 2. Chunks 1-3 produce byte-exact output, so the next step is a model-state comparison at a CHUNK boundary rather than a block boundary.

What is missing to do that: the real engine does NOT copy per chunk the way this port does. A watchpoint on the output buffer at a chunk offset fires inside a single large memcpy near the end of the block (eip 0x080dbe16 / 0x080dbe31), so it checkpoints the end of the block, not the chunk — comparing that against our chunk-4 dump gives a meaningless 44867-byte difference. The real chunk-loop address is the one thing still needed. NZO2_DUMP_CHUNK=<prefix> already dumps mem_ at every 0x8000 chunk start, ready for the comparison.

Also note: single-byte hardware watchpoints on the output buffer are unreliable here, because a data watchpoint only fires when the value CHANGES and the buffer often already holds the same byte. Watch a u32 instead.

Measuring gotcha: the traced position is a RING position, and the ring is NOT contiguous with file offsets across a wrap, nor across a block carrying a param2 layer. Validating match distances against file offsets naively produces false failures.

2. Latent, unverified

  • 0xd / 0xe chunk nibbles are reachable but never observed in a 71-file plus 90-file sweep.
  • decr_param == 3 never occurred in 654 blocks across 357 archives.
  • tt bit 0x20 is wired for -cc but no corpus archive sets it.
  • The -cD stored-chunk model reset (7a4d4c5) is scoped by measured behaviour, not decompile evidence.
  • A stored LZ block (decr_param == 1, param6 == 0, 7f07b6d) has no synthetic reproduction.
  • A mid-stream checksum record (aedeae9) is handled in the CM path only, single-entry archives only.
  • FUN_0809bdc0's initial table (85d046e) is a rule derived from the state its builder leaves, since FUN_080bd760 has no decompile either. It reproduces the binary's tables exactly for sel = 0, 1 and 31 and every archive tried, but it was not read out of the builder's code.

Closed: the decr_param == 2 audio defect, in full

It was the largest item on this page for several sessions and it turned out to be three separate things, none of them a missing algorithm in the sense first assumed:

what codecs affected fix
the inter-channel stage is selected by context bit 4 -co f54131b — wire FUN_08096e20 (already in-tree as LmsObject)
every configurable constant was hardcoded to -cO's value -cc, -co 113326b — per-codec orders and stereo parameter
the per-channel bit-count decoder is a different CLASS -co 85d046e — port FUN_0809bdc0 from raw disassembly

Per-codec constants, for reference:

constant source field -co -cO -cc
six predictor orders, one per pair plane obj +0x1c08 64, 8, 8 96, 8, 8 384, 16, 8
inter-channel order parameter obj+0xa870 +0x1404/+0x2814 4 8 16
context flag byte at FUN_080a5330 entry 0x13 0x03 0x0f
bit-count decoder vtable obj+0x38700 0x0813c860 0x0813c848 0x0813c848

Method notes worth keeping

  • Measure every codec on ONE corpus before choosing work. Doing that is what revealed -cD sitting 22 points below its sibling; comparing numbers across differently-generated corpora had hidden it.
  • A file failing under many codecs means a shared code path, not inherent difficulty. The four-file "hard set" turned out to be one defect in -cf/-cF and a different single defect in -cd/-cD.
  • "Right framing, wrong content" does not distinguish a missing core from a misconfigured one. That inference produced two wrong labels in the notes and cost several sessions.
  • Closing a coverage gap routinely exposes latent UB in newly-reached code — re-run ASAN after every gap fix, not only after touching a file you sanitised before. It has happened four times.
  • Fuzz the decoder on corrupt input. Byte-exactness testing cannot find memory-safety bugs; single-byte corruption under ASAN found an out-of-bounds heap write.
  • Verify a stored diagnosis before acting on it — the cheapest step is often a diff. The top item on this list for a whole session was "port the order-32 predictor, start by diffing against the validated ord32.h". Running that diff took one command and returned empty: the file was already in the tree. A recorded next-step can be stale or simply wrong; re-establish it before spending a session on it.
  • Group failures by WHERE they diverge, not by which file they are. Six unrelated-looking -cD failures (an m4a, a .adf, an .EXE, a module, a PHP source, a bitmap) all diverged at or within a few bytes of a 32768-byte chunk boundary, and in every case the preceding chunk was one that BYPASSES the codec's model. That single measurement — cmp offset vs. the chunk table — reframed the work from "port a predictor" to "find the dropped cross-chunk state", and it is a two-minute measurement.
  • Out-of-distribution output bytes name the guilty stage. In smooth audio-like data, emitting 01 03 02 06 where 95 94 91 95 belongs is not a wrong match copy (that would emit plausible bytes from elsewhere) — it is a predictor fed a wrong base. Read the values, not just the offsets.
  • Trace the primitive's inputs, not only its outputs. The -cD MTF bug was unreadable from output bytes alone, and obvious the moment the trace printed ctx=3a rank=1d -> sym=3a: the expected symbol was the rank code, which is what a freshly initialised table returns. One extra field in a printf.
  • "State persists across chunks" and "state resets every chunk" can both be wrong. For the nz_lzhds MTF table the rule is conditional — persist across token chunks and prefilter sub-chunks, reset across a full stored chunk. Test the reset scope in BOTH directions: an over-broad reset and an under-broad one both fail, on different files.
  • Any codec that is byte-exact on a file is an oracle for every codec that is not. With -cf/-cF/ -cd/-cD now clean on all 60 corpus files, every -co/-cO/-cc failure has four independent byte-exact references for the same plaintext.
  • Classify every failure before choosing which to fix. Reading one trace line per failing file (decr_param, and whether the sequence completed) turned "6 files in a wrong-bytes class + 2 synthetic audio failures" into "one audio defect accounting for 9 of 11 failures, plus two singletons". That changes what to work on next, and it cost about ten minutes.
  • A reference implementation reading an uninitialised field is not a specification. NZ.cpp takes an LZ block's output size from size18, which is only assigned under if (param6) — so at param6 == 0 it is UB. When the reference is silent or undefined, make the sizes add up against a real file instead.
  • Before reading a memory diff as model drift, classify every differing word. On the last open failure, "580 differing bytes" turned out to be mostly pointer fields baked into a captured cold blob plus a per-slot statistic counter. Two value pairs that looked exactly like adapted probabilities (63287 vs 62804) were the high halves of two heap addresses. Filtering pointer-shaped words first turns a 580-byte mystery into two unreferenced counters.
  • A data watchpoint only fires when the value CHANGES. A single-byte watch on an output buffer silently never triggers when the buffer already holds that byte. Watch a u32.
  • A port-invented address is not evidence about the original. This port appends the tier2 align table past the real object's model area (kTier2AlignOff == kMemSize) because nobody found the real one. The binary therefore reads 0 there, which looks exactly like "our cold value is wrong" and is nothing of the kind. Before treating a memory diff as a finding, check that the address means the same thing on both sides. I shipped nothing, but I did believe it for a while.
  • Run the suite before believing a measured fix. The align-table change had a clean measurement behind it and still dropped -cO from 11/11 to 7/11. The measurement was real; the inference from it was not.
  • The architecture notes can be wrong about code nobody read. FUN_0809bdc0 was on record as a "Fenwick-tree cumulative frequency" coder, inferred from its encoder counterpart. Reading the 207 instructions shows a linear cumulative scan in groups of four and an exact sliding window. A description derived by symmetry from a sibling function is a hypothesis, not a finding — and this one was labelled "presumed" in the notes, which is exactly the right way to have written it.
  • When a builder has no decompile, derive its output rule from the state it leaves. FUN_080bd760 could not be read, so the initial frequency tables came from reading the live object and reducing it: a ramp centred on 2*sel, the rest of the ring k % 64, and freq[s] = 1 + 8 * ringcount[s]. The rule proved itself by making the totals land on exactly 0x4000, the coder's own precision.
  • A hang is a diagnosis. The first attempt spun forever, which was the model being wrong in a specific way: a zero symbol frequency zeroes the range and the renormalize loop cannot make progress. Guarding it turned a hang into a clean decline and pointed straight at the initial table.
  • A byte-exact decode can still be declined by your own gate. -cc's last corpus failure was not a decode bug at all: the output matched the original byte for byte and the checksum-verification gate rejected it for lack of a checksum it could not locate. Check whether the DECODE is wrong before investigating the decoder — dump the pre-checksum output and diff it first, it costs one command.
  • A correct decode is its own oracle for locating metadata. Knowing the output let me compute the checksum the entry must carry and then search the archive for those four bytes; they were behind a 0x45 tag 971622 bytes in, which named the container shape immediately. Far faster than reading the parser.
  • When a port has hardcoded constants, suspect they are per-codec before suspecting the algorithm. The audio defect looked like a missing algorithm for months of session-time. It was three constants, all frozen at the one codec's values that happened to work. "Codec X is byte-exact and Y is not, on shared code" is much more often a parameter than a code path.
  • Instrument at the same points your ground-truth captures were taken. Dumping the port's arrays at the bit-count / post-residual-decode / per-predictor-call boundaries turned one opaque end-to-end failure into a three-row table that named the guilty stage immediately.
  • A hand-derivation that contradicts a measurement is wrong, even twice in a row. Two careful derivations from the decompile said the factor-update polarity and the shift convention had to change; per-stage measurement refuted both, and the raw assembly then confirmed the measurement. Ghidra's choice of which register is which operand is a guess, and cmp %esi,%eax; jg can render either way.
  • When a fix has no synthetic reproduction, say so in the commit. Eight constructed shapes failed to make the encoder emit a stored LZ block; recording that stops the next session from repeating the attempts and stops anyone from believing the suite guards it.

Clone this wiki locally