Skip to content

Component Status

Jump to bottom
YadeWira edited this page Sep 1, 2026 · 36 revisions

Component status

Detailed per-codec native-decode status. See the README for the measured summary table and quick start; see Changelog for the day-by-day history of how each status below was reached.

Known gaps where the bridge is invoked at runtime

(The bridge produces byte-exact output but the C++ does not decode these paths natively.)

  • lzpf stereo prefilter (-cf AND -cF) is now native end-to-end (see the component table below). The remaining -cF bridge case is the non-prefilter LZ77 variant-B path on some inputs.
  • lzhd (-cd/-cD): the coroutine token-LZ is native incl. multi-chunk text, large multi-stream files, and parallel containers (see the component table below). The LZ window is a single per-archive ring (single-container) or a fresh per-stream ring (parallel container) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB; modular helpers so non-power-of-two sizes work) — confirmed by GDB on FUN_08099050 (obj+0x978): 1/3/19/46 × 64 KB for text50/source.cpp/big_code/repeat_3M. The ring is sized to hold the whole compact recon, so its cursor advances monotonically and never wraps for real archives; it persists across the archive's 1 MB output streams. -cD (nz_lzhds) shares this entire front end and adds its own literal model — a per-context MTF rank table + adaptive order-N linear predictor, fed by an MSB-first bit reader + Exp-Golomb integer decoder (nz_lzhds.{h,cpp}) — now ported and native. The decoder still self-verifies against the stored per-file checksum and bridges on any mismatch (defense in depth). Remaining bridge cases: rare CM/BWT sub-chunks (shared by -cd/-cD).
  • optimum1 (-co) single-container decr_param==1 (LZ/CM) blocks are now native (see the component table below); an older theory that large -co/-cO used an unreversed "virtual-stream" DecLZ framing was disproven by GDB tracing against the real binary — the actual blocker was that DecLZ (a port of the community reference decoder) is simply never called by the real binary at all. The parallel multi-stream container (flag 0x0f, >~8 MB) is also native now for BOTH -co and -cO, reusing the same chunk-record format as -cf/-cd's parallel containers. optimum2 (-cO) single-container decr_param==1 blocks are native too now (its richer 8-context literal mixer + LZP secondary predictor ported into nz_optimum2_lz.{h,cpp}). Single-container archives with more than one stream_tag segment ("chain" mode — see the roadmap below) are also native now for both. Still bridging for both -co and -cO: decr_param==0 (BWT) blocks.
  • All five text-transform bits the encoder actually emits are now native: 0x10 (tt16 numbers), 0x08 (word dictionary), 0x04 (HTML closing-tag restoration), 0x02 (insert-LF), 0x20 (escape+RLE) and 0x01 (CR/CRLF). dece (the x86 exe-filter) is now ported too (648df9e, state model corrected in 491a54d), so the entire post-filter chain is native: param2, param1, all six text-transform bits, and dece. Note dece filter state persists across a run of consecutive dece blocks and resets when a non-dece block intervenes — the reference resets per block, which is wrong on ~25% of real dece archives. param1 (AddBytes) is now ported and working for -cc/-co/-cO — a real-world corpus sweep found and fixed a latent word-boundary truncation bug in its BitReader that had silently made it fail on real files whose side-stream length wasn't a multiple of 4 bytes.

Roadmap to 100% native decode

  1. ✅ Done: NZ_NO_BRIDGE=1 flag — when set, FindLegacyBackend* return empty and a missing native path is a hard error (no silent $PATH//usr/bin/nz fallback). This is what native_only_v2.sh uses to measure honestly.
  2. ✅ Done: the -cd cross-chunk / cross-stream LZ window. It is a single per-archive ring of size round(total_output / 0x10000) · 0x10000 (min 64 KB; GDB-confirmed on FUN_08099050 obj+0x978) that persists across the archive's 1 MB output streams (NzCdDecodeStream threads the ring position and the file-absolute output offset). The ring is sized to hold the whole compact recon, so it never wraps for real archives. This made multi-chunk text and large multi-stream files (a 7-chunk source file; 2–3.5 MB code/text/base64 incl. heavily-repetitive input) byte-exact native, with a checksum self-verify → bridge fallback as defense in depth.
  3. ✅ Done (-co/-cO, single-container AND parallel-container): ported the real linux32 -co/-cO LZ/CM cores (nz_optimum_lz.{h,cpp}, nz_optimum2_lz.{h,cpp}), LZMA-shaped LZ77 engines with 4 repeat-offset slots — not DecLZ, which is never called by the real binary. Also ✅ done: the tt_flags & 0x02 (InsertLF) text-transform bit for both -co and -cc; and param1 (AddBytesFilter) for -cc/-co/-cO. Remaining optimum work:
    • ✅ Done: decr_param==0 (BWT) for both -co and -cO, in both shapes. The block header layout was missing entirely (a non-CM decr_param==0 block carries param7 when param6 is set, a u32 inverse-BWT start position, and params 14/15 — none of which exist in the LZ layout). With param6==0 there is no entropy layer at all: the encoder found the BWT output incompressible and stored it raw, so the block expands to exactly its payload size and only the inverse BWT (NzBwtUntransform) runs. With param6==1 the output is wrapped in 256 independent per-leading-symbol buckets, each an arithmetic-coded move-to-front rank stream plus an optional byte-wise RLE expansion (NzBwtDecodeInput). New files nz_bwt.{h,cpp}. native_only_v2 -co/-cO 7/10 → 9/10 each, TOTAL 71 → 75/80; real-world 52-file corpus -co 29→32, -cO 27→31.
    • A compiler-visible latent bug inherited from the reference, surfaced by the above. The reference's rank-insertion loops read one element past their own uint8 P[256]: for (; new_c >= C[P[k + 8]] && k != 249; k += 8, new_c += 8). At k == 249 the left operand reads P[257], and the value cannot change control flow (the k check ends the loop that same iteration) — so it looks harmless, and is at -O0/-O1. At -O2 gcc is entitled to assume P[k+8] is in bounds, hence k <= 247, hence k != 249 is always true, and to delete the bound. The loop then runs away (k observed reaching 2313) and P[k] = last_rle writes far past P[255], corrupting the C[] array the compiler had placed immediately after P — surfacing much later as an absurd num_rle and a declined block. Testing the bound before the read removes the UB and restores the intended semantics. Found with ASAN (which flagged only the read, since at -O1 it does not exploit the UB) plus an A/B revert: with the reference form a real 1.5 MB BWT block fails, with the bound-first form it decodes byte-exact. The reference decoder is presumably miscompiled the same way at -O2, which may be part of why it is known to get some -co/-cO edges wrong.
    • ✅ Done: BWT param14/param15 follow-on transforms (NzBwtParam14/NzBwtParam15, ported from DecodeLZ_Param14 at NZ_LZ.cpp:543 and DecodeParam15 at NZ.cpp:843). Both run after the inverse BWT and before the shared post-filter chain, and both are LZ77 passes that find matches by scanning the byte stream for a two-byte escape tag instead of coding literal/match flags: param14 uses 0xfe 0xf1 + a selector byte with offsets relative to the output position and four repeat-offset slots; param15 uses 0xfe 0xf0 and names its source as an absolute offset (four raw big-endian one's-complement bytes taken from the byte stream) into the whole accumulated output stream, so a match can reach back into earlier blocks. param14 turned out to be common, not an edge case — roughly 20 of 220 sampled real files carry one, across audio/music/image formats. Note this param14 is NOT nz_cd_tokens.cpp's NzCdParam14 (the -cd char-class space-insertion transform) — same name, different algorithm.
    • ✅ Done, and the more interesting half: non-LZ block output must be fed into the LZ window. The first real file to get past the new param14 gate still failed — on a later, ordinary LZ block. In the original the LZ window is the shared accumulated-block buffer that every block writing into it advances (mem->data += size), which includes a BWT block's post-param14/15 output; only audio blocks are exempt, because DecodeFromStream returns before touching the window for those. This port keeps a private ring that only DecodeBlock ever wrote, so a later LZ match reaching back into a BWT block's output read stale ring bytes. Fixed with FeedWindow on both optimum decoders, called with the block's pre-post-filter bytes (which is what the window carries). Same class of bug as the CM stored-block desync fixed in 25d2f75. Shipped 296ee88. Measured on one identical 60-file real corpus with a baseline binary built at caeb56d: -co 43→52, -cO 45→54 (+9 each, 72%→88%) — the largest single jump of any fix in this project.
    • ✅ Done: single-container "chain" mode (multiple stream_tag segments in one archive, directly concatenated with no separator). A real 60 KB .doc file surfaced this — its first segment covered only 10229 of 60416 declared output bytes, and the very next byte was a second, independently valid stream_tag varint. An earlier session's negative "chain mode doesn't exist for -co" conclusion was based on insufficient fixtures (large synthetic files only, never a small real multi-block document). Fixed by looping over segments with one persistent decoder shared across the whole entry. Real-corpus sweep: -co 20→26/56, -cO 21→27/56.
  4. Port the remaining rare post-filter (dece exe-filter) and the CM stereo-audio variant.
  5. Partly done — and it turned out to be two unrelated gaps, not one. NanoZip has two distinct audio bitstream families:
    • decr_param==2 blocks (-cc/-co/-cO) — decoded by the reference's AudioPred, now ported as nz_audio.{h,cpp}. These blocks also have their own truncated header shape (a mode2_type byte, param6 forced to 1, size18, then STOP — no staged-checksum count and none of the param2/param1/param16/tt/dece fields), which is why every audio-bearing archive previously died before its first block trace: the ordinary parser read mode2_type as param6 and walked into the next record. decr_param==3 shares that shape (and, per a 357-archive sweep, never actually occurs). ✅ -cO is byte-exact on stereo and mono at 8/16/24-bit and on a 6 MB multi-block file exercising mode2_type=1 plus cross-block predictor state; native_only_v2 -cO 9/10 → 10/10, TOTAL 75 → 76/80. ❌ -co and -cc still decline — their output is bit-identical to the reference decoder's on the same payload (verified by building the reference AudioPred standalone), so the port is faithful and the reference itself diverges from the real binary on the shapes those two emit. The discriminator is which linear predictors the encoder enables: -cO's working set is {0,1,3}, -co adds 4, -cc adds 2 and 5. Sweeping the third pair's order over 13 values does not fix it → needs GDB against linux32/nz.
    • -cd/-cD audio — a completely different family: it surfaces as a chunk-header varint 0x0c, i.e. the same construct as the already-ported -cf/-cF audio block (0x04) but with bit 3 set, which selects real-binary FUN_080a9ca0 instead of the ported FUN_080a5bb0. AudioPred is not the right decoder here (its 2-byte-length-prefixed bit-count framing does not match). Unported.
  6. Extend real stereo/multichannel audio decode to -cd/-cD — a real-world corpus sweep found these codecs have near-zero coverage on real audio content (unlike -cf/-cF, whose stereo prefilter was fixed to 9/9 on the same corpus). These codecs don't have a dedicated prefilter like -cf/-cF — the gap is presumably in their own entropy/CM paths or the shared decr_param==0/BWT path.
  7. ✅ Done: integrty.doc declining under -cc was initially misattributed to param2's NzBwtRleDecodeU32 RLE expander (the function that returned false), but tracing one level upstream showed param2 was correct — the CM chunk feeding it had already produced garbage output partway through. Root cause: when a decr_param==0 chain has a param6==0 STORED (raw) chunk sitting between two CM-coded chunks, the CM model's rolling byte-window/hash context must still observe those stored bytes to stay in sync with the true output position, even though they were never CM-arithmetic-coded — the port never did this. Confirmed by building the community reference decoder standalone and reproducing the identical corruption on the real archive's exact bytes, then confirming the fix (feed stored bytes through a model-update-only path, NzCmFeedByte) against that same reproduction. Real-corpus sweep: -cc 31→32/56.

Component table

Component Cloned? Notes
CLI structure (l/t/x/a/s, all switches) ✅ 100% Full parser
Archive format (header, entry table, stream families 0x2b/0x3b/0x4b) ✅ 100%
Store (-cn) ✅ 100% Single-block stores are a contiguous tail copy. Files large enough to split (> ~0x30000) are stored as multiple blocks — [varint (len<<4)|0][raw] with a per-block checksum trailer between blocks — reassembled by TryAssembleStoredBlocks (-cn 9/10→10/10; the prior miss was a 215 KB source file stored as 196608 + 18566).
lzpf decode LZ77+arith path (-cf/-cF) ✅ native (10/10 measured) LZ77+arith (variants A 13-bit hash, B 24-bit hash) + literal + raw-bytecode block modes native. Native byte-exact on most real inputs under NZ_NO_BRIDGE=1. Large single-container multi-stream -cf/-cF (1–8 MB) decode byte-exact: the output splits into ~1 MB compressor streams chained [stream_tag][data] with an inter-stream whole-output checksum record (tag 0x45/0x47/0x26); dict/hash state persists across all streams. The parallel multi-stream container (header flag 0x0f, >~8 MB, multi-threaded encoder) also decodes byte-exact: N independent nz_lzpf streams, each a group of chunk records (type-1 slice size, type-10 u32 output offset, type-5/7/6 slice checksum, one or more type-0 compressed chunks concatenated into the stream), each decoded and placed at its offset and required to tile the whole output, every byte slice-checksum-verified. The sliding-window dict capacity is a multiple of 64 KiB the encoder picks by its threading (one of (p1+1)·64 KiB, or total rounded down/up to 64 KiB / 128 KiB; GDB-confirmed on FUN_080b6bb0 obj+4), so the decoder tries each candidate and adopts the first whose decode matches the slice/whole-output checksum (an unmodelled window wrap is rejected, never emitted). Mixed-binary content (real-world tars with both text and incompressible/random data) also decodes byte-exact: literal blocks now backfill hash_table after every literal run (legacy FUN_080b6d90/FUN_080b6cf0 — sparse, every 101 bytes, or dense, every byte, selected by a bit in the block header; variant B additionally zeroes its byte_buffer_8k side-table per FUN_080b6c20). Without this the table's untouched init value (3) leaked into later f6/f8/medium-match copies whenever a match referenced a hash bucket only ever touched during a literal run — byte-correct through many blocks by chance, then producing sparse single-byte corruption the moment a genuinely-collided bucket got referenced. Root-caused by GDB-diffing the decoder's hash_table against ground truth at the exact block boundary where output first diverged. The "8/8 100%" figure is from the low-entropy coverage_matrix.sh fixture only and does NOT reflect real data — see the measured table in the README. Variant B hash init=3 fix (commit 049d041) closed the multi-file silent corruption.
lzpf prefilter+arith path (-cf) ✅ mono + stereo native FUN_080a5330 + LPC filter (FUN_08095d90) ported; mono AND stereo audio byte-exact. The stereo path was closed end-to-end (stereo_lms_cf.nz → stereo_lms.wav, both blocks byte-exact, -cf native_only_v2 9/10→10/10). The full FUN_080a5330 stereo flow is now reproduced in DecodePFBlock: residuals are PLANAR (ch1=[0,per_chan), ch2=[per_chan,2·per_chan)) decoded by two per-channel DecodeArithBuffer calls (each reads its own Huffman header — a single n_elems call mis-locates the side stream by ~1.4 KB); the predictor-init reads a leading bit G (iStack_50078, gates the inter-channel LMS) then per channel [active(1), order(3 if active)] (order = bits+8); per-channel LPC with two persistent predictor states (pred/pred2, threaded across blocks); the verified inter-channel LMS FUN_08096e20 (ApplyLmsInterChannel, regression test TestLmsInterChannel); and FUN_080a50c0 reconstruction (ReconstructStereoSamples: per-channel delta-integrate + L/R (channels==1) or mid/side (channels==2) interleave). -cF (lzpf B / nz_lzpf_large) stereo is also native: it uses the SAME FUN_080a4ea0 arith as -cf (not a vtable path), differing only in the LPC filter order — FUN_08095d90 obj+0x1c08 is 8-tap for variant B vs 4-tap for variant A (GDB-confirmed), so LpcPredictor is now N-tap (8-tap adapts every sample, 4-tap every other). -cF native_only_v2 8/10→9/10. A real file-format-sample sweep later found the single-fixture validation above had missed real bit-depth/channel variation (5/9 on a 9-file real stereo/multichannel WAV corpus spanning 2/6 channels, 8/16/24-bit): fixed 4 real bugs — tail-remainder bytes silently dropped instead of copied verbatim; the generic (non-int16-fast-path) reconstruction always did plain L/R interleave instead of mid/side for channels==2; the LMS per-object adaptation-shift bits were never read/applied; and LPC/LMS predictor state wasn't reset across a literal/LZ77 block sitting between prefilter blocks (GDB-confirmed on the real binary: state is exactly zero after every non-prefilter block, carried forward after every prefilter block, 16-for-16). Real-corpus audio sweep now 9/9 for both -cf and -cF.
lzhd decoder (-cd/-cD) ✅ LZ (both -cd raw-literal and -cD MTF+predictor literal models) + block-RLE + raw-store + pure-literal + exe + text-pipeline (param14/line-RLE/CRLF/word-dict) + multi-chunk text + large multi-stream files + parallel containers native; only rare CM/BWT sub-chunks bridge The real linux32 -cd is a coroutine token-LZ (NOT the reference DecLZ). Fully ported in nz_cd_tokens.{h,cpp} and validated byte-exact against the binary: token assembler (FUN_080aa070), reconstruction with trailing-literal flush (FUN_08099050), per-column RLE run-expander (FUN_080acb90, thr=1 for the LEN column / 0 for LIT·OFF), param14 text transform (FUN_080a0ff0), the bounded-varint header (FUN_080b1dc0), and the integrated chunk/stream/block decoders (NzCdDecodeLzChunk/NzCdDecodeStream/NzCdDecodeBlock). Columns/literals reuse DecodeArithBuffer. The reconstruction runs over a single per-archive ring (FUN_08099050, obj+0x978) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB) — GDB-confirmed across text50/source.cpp/big_code/repeat_3M = 1/3/19/46 × 64 KB. The size need not be a power of two, so the ring uses modular helpers (not a bitmask). The ring is sized to hold the whole compact recon, so its cursor (obj+0x980) advances monotonically and never wraps for real archives (the reset/wrap path is a safety fallback). Large files split their output into 1 MB streams; the ring is allocated once and persists across streams (NzCdDecodeStream threads ring_pos and the file-absolute output offset), so cross-stream matches resolve correctly. Wired into the extract dispatcher and BYTE-EXACT end-to-end under NZ_NO_BRIDGE=1 for: token-LZ (recon == file), block-RLE (flag &2, post-recon run-length re-expansion of collapsed zero-runs), raw-store (per-column b0&1==0 and flag-&1-clear literals = verbatim bytes), pure-literal (no LZ tokens — the whole window is one literal stream; generator picks this when size_field==0 or v2==0), exe (flag &4: a BCJ-style x86 E8/E9 call/jmp address un-transform, NzCdExeUnfilter), and the text pipeline (flag &8, FUN_080a3c90): param14 (NzCdParam14), line-RLE (FUN_080a2f20), CRLF EOL (FUN_080a19b0), and the word-dictionary transform (FUN_080a0a00 = the reference TransformText_1_Dictionary + dict/char-trait tables, in nz_cd_texttransform_dict.cpp). Verified on text (map.txt), binary (image.cat), an ELF (elf.bin, flags=3 block-RLE), an EXE (play.exe, flags=5), atoll (multi-chunk &8), word-dictionary text, multi-chunk text (text_50k/repeat_100k/CRLF), a 7-chunk source file (single-stream 192 KB ring), large multi-stream files (2–3.5 MB code/text/base64, incl. heavily-repetitive input), and a varied real corpus spanning images/audio/video/executables. As defense in depth TryDecodeLegacyLzhd self-verifies the decoded output against the archive's stored per-file checksum and returns false on mismatch, so any unforeseen edge falls through to a byte-exact bridge decode (no silent corruption, and NZ_NO_BRIDGE native-only is a provable correctness signal). The parallel multi-stream container (header flag 0x0f, >~8 MB, mirroring the -cf/-cF parallel format) also decodes byte-exact: N independent nz_cd streams, each with its own FRESH ring (unlike the single-container case, where one ring is shared across the whole archive — confirmed empirically: each parallel-encoder thread owns its own nz_cd instance), sized per-slice via the same round(slice_size/0x10000)·0x10000 formula; each stream's type-0 chunks are already-delimited raw DecLZ blocks (no concatenation needed, unlike lzpf), decoded one at a time into the slice and verified against its own per-slice checksum before being tiled into the assembled output. Verified byte-exact on parallel containers up to 20 MB (text, random, and mixed text+random content). -cD (nz_lzhds = "lzhd strong") shares this ENTIRE front end (same token assembler, same column decoding, same LZ match/rep-cache model — even the same FUN_08099050-equivalent recon loop, dispatched via a vtable slot) and differs only in the literal coder: a per-context MTF rank table (256 contexts × 64 bytes: 32-slot rank table + 32-byte presence bitmap, seeded to the identity permutation with bitmap bits 0–31 set, not all-zero) plus a 4-stage adaptive linear predictor (pred = (w3·h3+w2·h2+w0·h0+0x200+w1·h1)>>10, sign-updated toward the residual), fed by an MSB-first bit-reader + two-level Exp-Golomb integer decoder over a dedicated -cD-only bitstream (one raw length byte + that many raw bytes, sitting between the token assembler's extra-bits blob and the literal column in the chunk layout — GDB-confirmed, not documented anywhere in the archive format itself). Ported in nz_lzhds.{h,cpp} and wired into both the single-container and parallel-container -cD paths in TryDecodeLegacyLzhd. Remaining bridge cases: rare CM/BWT sub-chunks (shared with -cd, e.g. stereo audio). The old DecLZ (nz_lzhd.cpp, nzdec_v0) is the wrong format for linux32 -cd/-cD.
optimum decoder (-co/-cO) ✅ both native single-container AND parallel-container LZ / ⚠️ BWT still bridges Three RE sessions of GDB tracing against the real linux32/nz binary found the real -co/-cO decode core is not DecLZ (the community-reference codec that name was borrowed from is never called) — it is a distinct LZMA-shaped LZ77 engine with 4 repeat-offset slots (rep0–3), a unary/Elias-gamma-style length/distance code, and an adaptive literal mixer sharing its low-level binary range coder with nz_cm.cpp. -co (FUN_0809e600, the "compact" nz_optimum1 variant, 4-context literal mixer) ported in nz_optimum_lz.{h,cpp}; a real port bug (the literal mixer's context-C seed sign-extended a byte the real binary zero-extends before shifting) was found and fixed via GDB divergence-tracing against a real failing archive. Also ports NzTextTransformRle (tt_flags & 0x20) and NzTextTransformInsertLf (tt_flags & 0x02). Native for -co single-container AND parallel-container (flag 0x0f, >~8 MB — reuses -cf/-cd's already-solved chunk-record format verbatim) archives with decr_param==1 (LZ/CM) blocks; still bridges on decr_param==0 (BWT). -cO (FUN_080a5d90, the "large" nz_optimum2 variant) shares the same backbone but a materially richer 8-context literal mixer plus a rolling 3-byte-hash LZP-style secondary predictor with no -co analog — ported in nz_optimum2_lz.{h,cpp}, reusing nz_cm.cpp's own kLzModelLNext/CM_Input_Bit primitives and -co's own dumped tables directly (GDB-confirmed byte-identical, not re-embedded) rather than re-deriving them. Wired into TryDecodeLegacyOptimum for method_p0==6, checksum-gated identically to -co. Found and fixed 3 real port bugs via GDB ground-truth (a units-to-bytes conversion in the dispatch-bit's second APM stage, a missing counter-based addressing scheme in rep-slot selection, and a copy-paste wrong base address in the length decoder's extra-bits loop) plus a 4th, subtler divergence that only manifested once mixer/weight state moved past the cold-start case (needed a longer, real-world-shaped stress fixture to even surface). Native for -cO single-container AND parallel-container (flag 0x0f) archives with decr_param==1 blocks — the parallel-container branch is shared verbatim with -co (the decoder type is selected once via a small closure since it's fixed per-archive, not per-stream), no new RE needed. decr_param==0 (BWT) still bridges for both -co and -cO.
cm decoder (-cc) ✅ native byte-exact Native CM decoder (NZ_CM.cpp, 1100 LOC). The documented "byte-26" divergence is FIXED (2026-06-08): it was a one-line port error in CM_Input_Bit — factors0_err used truncating factors[0] / 16 instead of the reference arithmetic shift factors[0] >> 4 (differs for negative values, flipping the factor[7] zeroing condition). Found by a per-bit next_probability diff vs a compiled reference oracle. With the full post-filter pipeline ported — param2 RLE (NzBwtRleDecodeU32), param1 AddBytes delta filter (NzAddBytesFilter), tt08 dictionary, tt16 number-transform (NzTextTransformNumber) — plus multi-chunk decoding (CM state persists across consecutive type-0 chunks), stored blocks (param6==0 = raw payload, for incompressible data), and NzTextTransformInsertLf (tt_flags & 0x02, shared with -co), -cc decodes byte-exact natively (no bridge) on source, numbers, dates, IPs, prose, markdown, random/large multi-chunk: 13/13 on a comprehensive sweep, 9/10 on native_only_v2 (the one miss is the deferred stereo audio-CM variant). NzAddBytesFilter's BitReader had a latent word-boundary truncation bug (rounded its side-stream length down to a multiple of 4 bytes instead of using the exact byte count, matching the reference exactly except for this one truncation) that a real-world corpus sweep found — it silently failed (clean decline, never wrong bytes) on any real file whose param1 side-stream length wasn't a multiple of 4, undetected by the synthetic test corpus; fixed, and the fix transparently benefits -co/-cO too since they call the same function. Cross-chunk continuation across a stored block was also missing: when a param6==0 stored (raw) chunk sits between two decr_param==0 CM chunks, the CM model must observe those stored bytes (model-update only, via NzCmFeedByte) to stay in sync with the true output position for the next CM chunk — found on a real file (integrty.doc), fixed, -cc real-corpus 31→32/56.
Encode for all methods ⚠️ functional Native BWT/store/literal writers; the legacy compression bridge was disabled in commit 049d041 (IsInternalLegacyCompressionBridgeCompressor returns false unconditionally). Codecs the native encoder cannot handle now produce an explicit error. All 8 methods byte-exact round-trip via the native encoder for low-entropy inputs.

Fixture-based benchmark (coverage_matrix.sh)

The coverage_matrix.sh test uses a deterministic AES-CTR-of-zeros fixture (low entropy). On that fixture:

native_strict_percent = 100% (8/8) — no bridge, no compat, no original binary subprocess.

Low-entropy input rarely triggers the prefilter+arith block mode, so the fixture passes fully native even though that path is not yet ported. Real-world compressible data (text, source code, binaries) will hit that path and fall back to the extract bridge for -cf/-cF.

Important caveat: native_strict_percent measures "no [bridge] / [compat] log line in stdout". It does NOT measure "the C++ code path actually ran the native decode vs silently called the legacy binary via FindLegacyBackend". A fixture that triggers the prefilter+arith block mode shows the bridge being invoked at runtime but still reports native_strict_percent=100 because the bridge produces byte-exact output. To verify the code path is truly native, use tests/native_only_v2.sh (NZ_NO_BRIDGE=1) — see the README's measured table, which is the honest number.

Real-corpus standing (60 files, 2026-09-01, all measured on the same corpus)

codec pass codec pass
-cn 60/60 -cd 60/60
-cf 60/60 -cD 60/60
-cF 60/60 -cc 60/60
-co 60/60 -cO 59/60

479/480 overall. Seven of the eight codecs are perfect on this corpus; one file remains. All five codecs that are perfect here — -cn, -cf, -cF, -cd, -cD — cover the whole 0x2b family plus store. Every remaining failure is in -co/-cO/-cc.

(2026-09-01: -cD stage-index fix c831c31, -cf/-cF exe-filter fix f885e1c, prefilter hash-backfill fix eb66e8c, prefilter sub-chunk cd5e3f3, and the two cross-chunk state fixes 7a4d4c5 that took -cd 58 → 60 and -cD 52 → 60.)

Measuring every codec on ONE corpus is what surfaced the -cD deficit (then 34/60); comparing numbers across differently-generated corpora had been hiding it. Four files (Moly, summer.php, M05.AMF, RESOURCE.001) fail across most codecs and are the shared hard set.

Open work (as of 2026-09-01, after 85d046e)

Synthetic suite: 88/88 — 100%. All eight codecs at 11/11, zero bridge, no fixture needs the legacy binary for any method. Real corpus (60 files, all codecs on the same corpus): -cn 60 · -cf 60 · -cF 60 · -cd 60 · -cD 60 · -co 60 · -cc 60 · -cO 59 — 479/480.

One failure is left in the whole project. There is no unported function, block kind, chunk kind or post-filter remaining.

1. -cO on doc/Fonts Poster-color.dp — one mis-decoded literal bit

Fully localised, so this is a warm start rather than an investigation:

  • Block 1 (19219 bytes) is byte-exact. Block 2 is 122944 bytes, decoded as four 0x8000 chunks.
  • Chunks 1-3 (ring 19219..117522, 98304 bytes) are completely correct: 91439 literals, zero wrong.
  • The ring then wraps (EnsureHeadroom, cursor → 0). Chunk 4 stays correct for 2922 more bytes and 59 further matches, then emits one wrong literal byte at chunk-4 ring position 2923 (file offset 122018): 0x80 where 0x40 belongs — a single bit in the literal bit-tree.
  • Everything after is downstream damage. The visible error is FAIL@distance pos=10568 acc=385363 capacity=131072 about 7.6k later: a bogus slot 18, where the slot ladder covers distances (2^s, 2^(s+1)] so a 2^17 window admits at most slot 16.

A latent adaptive-model state divergence that only flipped a decision after ~101k correct bytes, not a structural bug at the wrap. The wrap is still the first thing to re-examine: EnsureHeadroom memsets [cursor, cap+256) on the first scroll and then copies the last 256 bytes to base-256; if the real code copies BEFORE zeroing, the 256-byte backward guard differs.

Measuring gotcha, read this first: the traced position is a RING position, and the ring is NOT contiguous with file offsets across a wrap, nor across a block carrying a param2 layer (the ring holds pre-post-filter output). Validating match distances against file offsets naively produces false failures — it reported a bogus "first invalid match at pos=608" that was in fact perfectly correct once mapped properly.

2. Latent, unverified

  • 0xd / 0xe chunk nibbles are reachable but never observed in a 71-file plus 90-file sweep.
  • decr_param == 3 never occurred in 654 blocks across 357 archives.
  • tt bit 0x20 is wired for -cc but no corpus archive sets it.
  • The -cD stored-chunk model reset (7a4d4c5) is scoped by measured behaviour, not decompile evidence.
  • A stored LZ block (decr_param == 1, param6 == 0, 7f07b6d) has no synthetic reproduction.
  • A mid-stream checksum record (aedeae9) is handled in the CM path only, single-entry archives only.
  • FUN_0809bdc0's initial table (85d046e) is a rule derived from the state its builder leaves, since FUN_080bd760 has no decompile either. It reproduces the binary's tables exactly for sel = 0, 1 and 31 and every archive tried, but it was not read out of the builder's code.

Closed: the decr_param == 2 audio defect, in full

It was the largest item on this page for several sessions and it turned out to be three separate things, none of them a missing algorithm in the sense first assumed:

what codecs affected fix
the inter-channel stage is selected by context bit 4 -co f54131b — wire FUN_08096e20 (already in-tree as LmsObject)
every configurable constant was hardcoded to -cO's value -cc, -co 113326b — per-codec orders and stereo parameter
the per-channel bit-count decoder is a different CLASS -co 85d046e — port FUN_0809bdc0 from raw disassembly

Per-codec constants, for reference:

constant source field -co -cO -cc
six predictor orders, one per pair plane obj +0x1c08 64, 8, 8 96, 8, 8 384, 16, 8
inter-channel order parameter obj+0xa870 +0x1404/+0x2814 4 8 16
context flag byte at FUN_080a5330 entry 0x13 0x03 0x0f
bit-count decoder vtable obj+0x38700 0x0813c860 0x0813c848 0x0813c848

Method notes worth keeping

  • Measure every codec on ONE corpus before choosing work. Doing that is what revealed -cD sitting 22 points below its sibling; comparing numbers across differently-generated corpora had hidden it.
  • A file failing under many codecs means a shared code path, not inherent difficulty. The four-file "hard set" turned out to be one defect in -cf/-cF and a different single defect in -cd/-cD.
  • "Right framing, wrong content" does not distinguish a missing core from a misconfigured one. That inference produced two wrong labels in the notes and cost several sessions.
  • Closing a coverage gap routinely exposes latent UB in newly-reached code — re-run ASAN after every gap fix, not only after touching a file you sanitised before. It has happened four times.
  • Fuzz the decoder on corrupt input. Byte-exactness testing cannot find memory-safety bugs; single-byte corruption under ASAN found an out-of-bounds heap write.
  • Verify a stored diagnosis before acting on it — the cheapest step is often a diff. The top item on this list for a whole session was "port the order-32 predictor, start by diffing against the validated ord32.h". Running that diff took one command and returned empty: the file was already in the tree. A recorded next-step can be stale or simply wrong; re-establish it before spending a session on it.
  • Group failures by WHERE they diverge, not by which file they are. Six unrelated-looking -cD failures (an m4a, a .adf, an .EXE, a module, a PHP source, a bitmap) all diverged at or within a few bytes of a 32768-byte chunk boundary, and in every case the preceding chunk was one that BYPASSES the codec's model. That single measurement — cmp offset vs. the chunk table — reframed the work from "port a predictor" to "find the dropped cross-chunk state", and it is a two-minute measurement.
  • Out-of-distribution output bytes name the guilty stage. In smooth audio-like data, emitting 01 03 02 06 where 95 94 91 95 belongs is not a wrong match copy (that would emit plausible bytes from elsewhere) — it is a predictor fed a wrong base. Read the values, not just the offsets.
  • Trace the primitive's inputs, not only its outputs. The -cD MTF bug was unreadable from output bytes alone, and obvious the moment the trace printed ctx=3a rank=1d -> sym=3a: the expected symbol was the rank code, which is what a freshly initialised table returns. One extra field in a printf.
  • "State persists across chunks" and "state resets every chunk" can both be wrong. For the nz_lzhds MTF table the rule is conditional — persist across token chunks and prefilter sub-chunks, reset across a full stored chunk. Test the reset scope in BOTH directions: an over-broad reset and an under-broad one both fail, on different files.
  • Any codec that is byte-exact on a file is an oracle for every codec that is not. With -cf/-cF/ -cd/-cD now clean on all 60 corpus files, every -co/-cO/-cc failure has four independent byte-exact references for the same plaintext.
  • Classify every failure before choosing which to fix. Reading one trace line per failing file (decr_param, and whether the sequence completed) turned "6 files in a wrong-bytes class + 2 synthetic audio failures" into "one audio defect accounting for 9 of 11 failures, plus two singletons". That changes what to work on next, and it cost about ten minutes.
  • A reference implementation reading an uninitialised field is not a specification. NZ.cpp takes an LZ block's output size from size18, which is only assigned under if (param6) — so at param6 == 0 it is UB. When the reference is silent or undefined, make the sizes add up against a real file instead.
  • The architecture notes can be wrong about code nobody read. FUN_0809bdc0 was on record as a "Fenwick-tree cumulative frequency" coder, inferred from its encoder counterpart. Reading the 207 instructions shows a linear cumulative scan in groups of four and an exact sliding window. A description derived by symmetry from a sibling function is a hypothesis, not a finding — and this one was labelled "presumed" in the notes, which is exactly the right way to have written it.
  • When a builder has no decompile, derive its output rule from the state it leaves. FUN_080bd760 could not be read, so the initial frequency tables came from reading the live object and reducing it: a ramp centred on 2*sel, the rest of the ring k % 64, and freq[s] = 1 + 8 * ringcount[s]. The rule proved itself by making the totals land on exactly 0x4000, the coder's own precision.
  • A hang is a diagnosis. The first attempt spun forever, which was the model being wrong in a specific way: a zero symbol frequency zeroes the range and the renormalize loop cannot make progress. Guarding it turned a hang into a clean decline and pointed straight at the initial table.
  • A byte-exact decode can still be declined by your own gate. -cc's last corpus failure was not a decode bug at all: the output matched the original byte for byte and the checksum-verification gate rejected it for lack of a checksum it could not locate. Check whether the DECODE is wrong before investigating the decoder — dump the pre-checksum output and diff it first, it costs one command.
  • A correct decode is its own oracle for locating metadata. Knowing the output let me compute the checksum the entry must carry and then search the archive for those four bytes; they were behind a 0x45 tag 971622 bytes in, which named the container shape immediately. Far faster than reading the parser.
  • When a port has hardcoded constants, suspect they are per-codec before suspecting the algorithm. The audio defect looked like a missing algorithm for months of session-time. It was three constants, all frozen at the one codec's values that happened to work. "Codec X is byte-exact and Y is not, on shared code" is much more often a parameter than a code path.
  • Instrument at the same points your ground-truth captures were taken. Dumping the port's arrays at the bit-count / post-residual-decode / per-predictor-call boundaries turned one opaque end-to-end failure into a three-row table that named the guilty stage immediately.
  • A hand-derivation that contradicts a measurement is wrong, even twice in a row. Two careful derivations from the decompile said the factor-update polarity and the shift convention had to change; per-stage measurement refuted both, and the raw assembly then confirmed the measurement. Ghidra's choice of which register is which operand is a guess, and cmp %esi,%eax; jg can render either way.
  • When a fix has no synthetic reproduction, say so in the commit. Eight constructed shapes failed to make the encoder emit a stored LZ block; recording that stops the next session from repeating the attempts and stops anyone from believing the suite guards it.

Clone this wiki locally