Skip to content

Component Status

Jump to bottom
YadeWira edited this page Sep 1, 2026 · 36 revisions

Component status

Detailed per-codec native-decode status. See the README for the measured summary table and quick start; see Changelog for the day-by-day history of how each status below was reached.

Known gaps where the bridge is invoked at runtime

(The bridge produces byte-exact output but the C++ does not decode these paths natively.)

  • lzpf stereo prefilter (-cf AND -cF) is now native end-to-end (see the component table below). The remaining -cF bridge case is the non-prefilter LZ77 variant-B path on some inputs.
  • lzhd (-cd/-cD): the coroutine token-LZ is native incl. multi-chunk text, large multi-stream files, and parallel containers (see the component table below). The LZ window is a single per-archive ring (single-container) or a fresh per-stream ring (parallel container) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB; modular helpers so non-power-of-two sizes work) — confirmed by GDB on FUN_08099050 (obj+0x978): 1/3/19/46 × 64 KB for text50/source.cpp/big_code/repeat_3M. The ring is sized to hold the whole compact recon, so its cursor advances monotonically and never wraps for real archives; it persists across the archive's 1 MB output streams. -cD (nz_lzhds) shares this entire front end and adds its own literal model — a per-context MTF rank table + adaptive order-N linear predictor, fed by an MSB-first bit reader + Exp-Golomb integer decoder (nz_lzhds.{h,cpp}) — now ported and native. The decoder still self-verifies against the stored per-file checksum and bridges on any mismatch (defense in depth). Remaining bridge cases: rare CM/BWT sub-chunks (shared by -cd/-cD).
  • optimum1 (-co) single-container decr_param==1 (LZ/CM) blocks are now native (see the component table below); an older theory that large -co/-cO used an unreversed "virtual-stream" DecLZ framing was disproven by GDB tracing against the real binary — the actual blocker was that DecLZ (a port of the community reference decoder) is simply never called by the real binary at all. The parallel multi-stream container (flag 0x0f, >~8 MB) is also native now for BOTH -co and -cO, reusing the same chunk-record format as -cf/-cd's parallel containers. optimum2 (-cO) single-container decr_param==1 blocks are native too now (its richer 8-context literal mixer + LZP secondary predictor ported into nz_optimum2_lz.{h,cpp}). Single-container archives with more than one stream_tag segment ("chain" mode — see the roadmap below) are also native now for both. Still bridging for both -co and -cO: decr_param==0 (BWT) blocks.
  • All five text-transform bits the encoder actually emits are now native: 0x10 (tt16 numbers), 0x08 (word dictionary), 0x04 (HTML closing-tag restoration), 0x02 (insert-LF), 0x20 (escape+RLE) and 0x01 (CR/CRLF). dece (the x86 exe-filter) is now ported too (648df9e, state model corrected in 491a54d), so the entire post-filter chain is native: param2, param1, all six text-transform bits, and dece. Note dece filter state persists across a run of consecutive dece blocks and resets when a non-dece block intervenes — the reference resets per block, which is wrong on ~25% of real dece archives. param1 (AddBytes) is now ported and working for -cc/-co/-cO — a real-world corpus sweep found and fixed a latent word-boundary truncation bug in its BitReader that had silently made it fail on real files whose side-stream length wasn't a multiple of 4 bytes.

Roadmap to 100% native decode

  1. ✅ Done: NZ_NO_BRIDGE=1 flag — when set, FindLegacyBackend* return empty and a missing native path is a hard error (no silent $PATH//usr/bin/nz fallback). This is what native_only_v2.sh uses to measure honestly.
  2. ✅ Done: the -cd cross-chunk / cross-stream LZ window. It is a single per-archive ring of size round(total_output / 0x10000) · 0x10000 (min 64 KB; GDB-confirmed on FUN_08099050 obj+0x978) that persists across the archive's 1 MB output streams (NzCdDecodeStream threads the ring position and the file-absolute output offset). The ring is sized to hold the whole compact recon, so it never wraps for real archives. This made multi-chunk text and large multi-stream files (a 7-chunk source file; 2–3.5 MB code/text/base64 incl. heavily-repetitive input) byte-exact native, with a checksum self-verify → bridge fallback as defense in depth.
  3. ✅ Done (-co/-cO, single-container AND parallel-container): ported the real linux32 -co/-cO LZ/CM cores (nz_optimum_lz.{h,cpp}, nz_optimum2_lz.{h,cpp}), LZMA-shaped LZ77 engines with 4 repeat-offset slots — not DecLZ, which is never called by the real binary. Also ✅ done: the tt_flags & 0x02 (InsertLF) text-transform bit for both -co and -cc; and param1 (AddBytesFilter) for -cc/-co/-cO. Remaining optimum work:
    • ✅ Done: decr_param==0 (BWT) for both -co and -cO, in both shapes. The block header layout was missing entirely (a non-CM decr_param==0 block carries param7 when param6 is set, a u32 inverse-BWT start position, and params 14/15 — none of which exist in the LZ layout). With param6==0 there is no entropy layer at all: the encoder found the BWT output incompressible and stored it raw, so the block expands to exactly its payload size and only the inverse BWT (NzBwtUntransform) runs. With param6==1 the output is wrapped in 256 independent per-leading-symbol buckets, each an arithmetic-coded move-to-front rank stream plus an optional byte-wise RLE expansion (NzBwtDecodeInput). New files nz_bwt.{h,cpp}. native_only_v2 -co/-cO 7/10 → 9/10 each, TOTAL 71 → 75/80; real-world 52-file corpus -co 29→32, -cO 27→31.
    • A compiler-visible latent bug inherited from the reference, surfaced by the above. The reference's rank-insertion loops read one element past their own uint8 P[256]: for (; new_c >= C[P[k + 8]] && k != 249; k += 8, new_c += 8). At k == 249 the left operand reads P[257], and the value cannot change control flow (the k check ends the loop that same iteration) — so it looks harmless, and is at -O0/-O1. At -O2 gcc is entitled to assume P[k+8] is in bounds, hence k <= 247, hence k != 249 is always true, and to delete the bound. The loop then runs away (k observed reaching 2313) and P[k] = last_rle writes far past P[255], corrupting the C[] array the compiler had placed immediately after P — surfacing much later as an absurd num_rle and a declined block. Testing the bound before the read removes the UB and restores the intended semantics. Found with ASAN (which flagged only the read, since at -O1 it does not exploit the UB) plus an A/B revert: with the reference form a real 1.5 MB BWT block fails, with the bound-first form it decodes byte-exact. The reference decoder is presumably miscompiled the same way at -O2, which may be part of why it is known to get some -co/-cO edges wrong.
    • ✅ Done: BWT param14/param15 follow-on transforms (NzBwtParam14/NzBwtParam15, ported from DecodeLZ_Param14 at NZ_LZ.cpp:543 and DecodeParam15 at NZ.cpp:843). Both run after the inverse BWT and before the shared post-filter chain, and both are LZ77 passes that find matches by scanning the byte stream for a two-byte escape tag instead of coding literal/match flags: param14 uses 0xfe 0xf1 + a selector byte with offsets relative to the output position and four repeat-offset slots; param15 uses 0xfe 0xf0 and names its source as an absolute offset (four raw big-endian one's-complement bytes taken from the byte stream) into the whole accumulated output stream, so a match can reach back into earlier blocks. param14 turned out to be common, not an edge case — roughly 20 of 220 sampled real files carry one, across audio/music/image formats. Note this param14 is NOT nz_cd_tokens.cpp's NzCdParam14 (the -cd char-class space-insertion transform) — same name, different algorithm.
    • ✅ Done, and the more interesting half: non-LZ block output must be fed into the LZ window. The first real file to get past the new param14 gate still failed — on a later, ordinary LZ block. In the original the LZ window is the shared accumulated-block buffer that every block writing into it advances (mem->data += size), which includes a BWT block's post-param14/15 output; only audio blocks are exempt, because DecodeFromStream returns before touching the window for those. This port keeps a private ring that only DecodeBlock ever wrote, so a later LZ match reaching back into a BWT block's output read stale ring bytes. Fixed with FeedWindow on both optimum decoders, called with the block's pre-post-filter bytes (which is what the window carries). Same class of bug as the CM stored-block desync fixed in 25d2f75. Shipped 296ee88. Measured on one identical 60-file real corpus with a baseline binary built at caeb56d: -co 43→52, -cO 45→54 (+9 each, 72%→88%) — the largest single jump of any fix in this project.
    • ✅ Done: single-container "chain" mode (multiple stream_tag segments in one archive, directly concatenated with no separator). A real 60 KB .doc file surfaced this — its first segment covered only 10229 of 60416 declared output bytes, and the very next byte was a second, independently valid stream_tag varint. An earlier session's negative "chain mode doesn't exist for -co" conclusion was based on insufficient fixtures (large synthetic files only, never a small real multi-block document). Fixed by looping over segments with one persistent decoder shared across the whole entry. Real-corpus sweep: -co 20→26/56, -cO 21→27/56.
  4. Port the remaining rare post-filter (dece exe-filter) and the CM stereo-audio variant.
  5. Partly done — and it turned out to be two unrelated gaps, not one. NanoZip has two distinct audio bitstream families:
    • decr_param==2 blocks (-cc/-co/-cO) — decoded by the reference's AudioPred, now ported as nz_audio.{h,cpp}. These blocks also have their own truncated header shape (a mode2_type byte, param6 forced to 1, size18, then STOP — no staged-checksum count and none of the param2/param1/param16/tt/dece fields), which is why every audio-bearing archive previously died before its first block trace: the ordinary parser read mode2_type as param6 and walked into the next record. decr_param==3 shares that shape (and, per a 357-archive sweep, never actually occurs). ✅ -cO is byte-exact on stereo and mono at 8/16/24-bit and on a 6 MB multi-block file exercising mode2_type=1 plus cross-block predictor state; native_only_v2 -cO 9/10 → 10/10, TOTAL 75 → 76/80. ❌ -co and -cc still decline — their output is bit-identical to the reference decoder's on the same payload (verified by building the reference AudioPred standalone), so the port is faithful and the reference itself diverges from the real binary on the shapes those two emit. The discriminator is which linear predictors the encoder enables: -cO's working set is {0,1,3}, -co adds 4, -cc adds 2 and 5. Sweeping the third pair's order over 13 values does not fix it → needs GDB against linux32/nz.
    • -cd/-cD audio — a completely different family: it surfaces as a chunk-header varint 0x0c, i.e. the same construct as the already-ported -cf/-cF audio block (0x04) but with bit 3 set, which selects real-binary FUN_080a9ca0 instead of the ported FUN_080a5bb0. AudioPred is not the right decoder here (its 2-byte-length-prefixed bit-count framing does not match). Unported.
  6. Extend real stereo/multichannel audio decode to -cd/-cD — a real-world corpus sweep found these codecs have near-zero coverage on real audio content (unlike -cf/-cF, whose stereo prefilter was fixed to 9/9 on the same corpus). These codecs don't have a dedicated prefilter like -cf/-cF — the gap is presumably in their own entropy/CM paths or the shared decr_param==0/BWT path.
  7. ✅ Done: integrty.doc declining under -cc was initially misattributed to param2's NzBwtRleDecodeU32 RLE expander (the function that returned false), but tracing one level upstream showed param2 was correct — the CM chunk feeding it had already produced garbage output partway through. Root cause: when a decr_param==0 chain has a param6==0 STORED (raw) chunk sitting between two CM-coded chunks, the CM model's rolling byte-window/hash context must still observe those stored bytes to stay in sync with the true output position, even though they were never CM-arithmetic-coded — the port never did this. Confirmed by building the community reference decoder standalone and reproducing the identical corruption on the real archive's exact bytes, then confirming the fix (feed stored bytes through a model-update-only path, NzCmFeedByte) against that same reproduction. Real-corpus sweep: -cc 31→32/56.

Component table

Component Cloned? Notes
CLI structure (l/t/x/a/s, all switches) ✅ 100% Full parser
Archive format (header, entry table, stream families 0x2b/0x3b/0x4b) ✅ 100%
Store (-cn) ✅ 100% Single-block stores are a contiguous tail copy. Files large enough to split (> ~0x30000) are stored as multiple blocks — [varint (len<<4)|0][raw] with a per-block checksum trailer between blocks — reassembled by TryAssembleStoredBlocks (-cn 9/10→10/10; the prior miss was a 215 KB source file stored as 196608 + 18566).
lzpf decode LZ77+arith path (-cf/-cF) ✅ native (10/10 measured) LZ77+arith (variants A 13-bit hash, B 24-bit hash) + literal + raw-bytecode block modes native. Native byte-exact on most real inputs under NZ_NO_BRIDGE=1. Large single-container multi-stream -cf/-cF (1–8 MB) decode byte-exact: the output splits into ~1 MB compressor streams chained [stream_tag][data] with an inter-stream whole-output checksum record (tag 0x45/0x47/0x26); dict/hash state persists across all streams. The parallel multi-stream container (header flag 0x0f, >~8 MB, multi-threaded encoder) also decodes byte-exact: N independent nz_lzpf streams, each a group of chunk records (type-1 slice size, type-10 u32 output offset, type-5/7/6 slice checksum, one or more type-0 compressed chunks concatenated into the stream), each decoded and placed at its offset and required to tile the whole output, every byte slice-checksum-verified. The sliding-window dict capacity is a multiple of 64 KiB the encoder picks by its threading (one of (p1+1)·64 KiB, or total rounded down/up to 64 KiB / 128 KiB; GDB-confirmed on FUN_080b6bb0 obj+4), so the decoder tries each candidate and adopts the first whose decode matches the slice/whole-output checksum (an unmodelled window wrap is rejected, never emitted). Mixed-binary content (real-world tars with both text and incompressible/random data) also decodes byte-exact: literal blocks now backfill hash_table after every literal run (legacy FUN_080b6d90/FUN_080b6cf0 — sparse, every 101 bytes, or dense, every byte, selected by a bit in the block header; variant B additionally zeroes its byte_buffer_8k side-table per FUN_080b6c20). Without this the table's untouched init value (3) leaked into later f6/f8/medium-match copies whenever a match referenced a hash bucket only ever touched during a literal run — byte-correct through many blocks by chance, then producing sparse single-byte corruption the moment a genuinely-collided bucket got referenced. Root-caused by GDB-diffing the decoder's hash_table against ground truth at the exact block boundary where output first diverged. The "8/8 100%" figure is from the low-entropy coverage_matrix.sh fixture only and does NOT reflect real data — see the measured table in the README. Variant B hash init=3 fix (commit 049d041) closed the multi-file silent corruption.
lzpf prefilter+arith path (-cf) ✅ mono + stereo native FUN_080a5330 + LPC filter (FUN_08095d90) ported; mono AND stereo audio byte-exact. The stereo path was closed end-to-end (stereo_lms_cf.nz → stereo_lms.wav, both blocks byte-exact, -cf native_only_v2 9/10→10/10). The full FUN_080a5330 stereo flow is now reproduced in DecodePFBlock: residuals are PLANAR (ch1=[0,per_chan), ch2=[per_chan,2·per_chan)) decoded by two per-channel DecodeArithBuffer calls (each reads its own Huffman header — a single n_elems call mis-locates the side stream by ~1.4 KB); the predictor-init reads a leading bit G (iStack_50078, gates the inter-channel LMS) then per channel [active(1), order(3 if active)] (order = bits+8); per-channel LPC with two persistent predictor states (pred/pred2, threaded across blocks); the verified inter-channel LMS FUN_08096e20 (ApplyLmsInterChannel, regression test TestLmsInterChannel); and FUN_080a50c0 reconstruction (ReconstructStereoSamples: per-channel delta-integrate + L/R (channels==1) or mid/side (channels==2) interleave). -cF (lzpf B / nz_lzpf_large) stereo is also native: it uses the SAME FUN_080a4ea0 arith as -cf (not a vtable path), differing only in the LPC filter order — FUN_08095d90 obj+0x1c08 is 8-tap for variant B vs 4-tap for variant A (GDB-confirmed), so LpcPredictor is now N-tap (8-tap adapts every sample, 4-tap every other). -cF native_only_v2 8/10→9/10. A real file-format-sample sweep later found the single-fixture validation above had missed real bit-depth/channel variation (5/9 on a 9-file real stereo/multichannel WAV corpus spanning 2/6 channels, 8/16/24-bit): fixed 4 real bugs — tail-remainder bytes silently dropped instead of copied verbatim; the generic (non-int16-fast-path) reconstruction always did plain L/R interleave instead of mid/side for channels==2; the LMS per-object adaptation-shift bits were never read/applied; and LPC/LMS predictor state wasn't reset across a literal/LZ77 block sitting between prefilter blocks (GDB-confirmed on the real binary: state is exactly zero after every non-prefilter block, carried forward after every prefilter block, 16-for-16). Real-corpus audio sweep now 9/9 for both -cf and -cF.
lzhd decoder (-cd/-cD) ✅ LZ (both -cd raw-literal and -cD MTF+predictor literal models) + block-RLE + raw-store + pure-literal + exe + text-pipeline (param14/line-RLE/CRLF/word-dict) + multi-chunk text + large multi-stream files + parallel containers native; only rare CM/BWT sub-chunks bridge The real linux32 -cd is a coroutine token-LZ (NOT the reference DecLZ). Fully ported in nz_cd_tokens.{h,cpp} and validated byte-exact against the binary: token assembler (FUN_080aa070), reconstruction with trailing-literal flush (FUN_08099050), per-column RLE run-expander (FUN_080acb90, thr=1 for the LEN column / 0 for LIT·OFF), param14 text transform (FUN_080a0ff0), the bounded-varint header (FUN_080b1dc0), and the integrated chunk/stream/block decoders (NzCdDecodeLzChunk/NzCdDecodeStream/NzCdDecodeBlock). Columns/literals reuse DecodeArithBuffer. The reconstruction runs over a single per-archive ring (FUN_08099050, obj+0x978) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB) — GDB-confirmed across text50/source.cpp/big_code/repeat_3M = 1/3/19/46 × 64 KB. The size need not be a power of two, so the ring uses modular helpers (not a bitmask). The ring is sized to hold the whole compact recon, so its cursor (obj+0x980) advances monotonically and never wraps for real archives (the reset/wrap path is a safety fallback). Large files split their output into 1 MB streams; the ring is allocated once and persists across streams (NzCdDecodeStream threads ring_pos and the file-absolute output offset), so cross-stream matches resolve correctly. Wired into the extract dispatcher and BYTE-EXACT end-to-end under NZ_NO_BRIDGE=1 for: token-LZ (recon == file), block-RLE (flag &2, post-recon run-length re-expansion of collapsed zero-runs), raw-store (per-column b0&1==0 and flag-&1-clear literals = verbatim bytes), pure-literal (no LZ tokens — the whole window is one literal stream; generator picks this when size_field==0 or v2==0), exe (flag &4: a BCJ-style x86 E8/E9 call/jmp address un-transform, NzCdExeUnfilter), and the text pipeline (flag &8, FUN_080a3c90): param14 (NzCdParam14), line-RLE (FUN_080a2f20), CRLF EOL (FUN_080a19b0), and the word-dictionary transform (FUN_080a0a00 = the reference TransformText_1_Dictionary + dict/char-trait tables, in nz_cd_texttransform_dict.cpp). Verified on text (map.txt), binary (image.cat), an ELF (elf.bin, flags=3 block-RLE), an EXE (play.exe, flags=5), atoll (multi-chunk &8), word-dictionary text, multi-chunk text (text_50k/repeat_100k/CRLF), a 7-chunk source file (single-stream 192 KB ring), large multi-stream files (2–3.5 MB code/text/base64, incl. heavily-repetitive input), and a varied real corpus spanning images/audio/video/executables. As defense in depth TryDecodeLegacyLzhd self-verifies the decoded output against the archive's stored per-file checksum and returns false on mismatch, so any unforeseen edge falls through to a byte-exact bridge decode (no silent corruption, and NZ_NO_BRIDGE native-only is a provable correctness signal). The parallel multi-stream container (header flag 0x0f, >~8 MB, mirroring the -cf/-cF parallel format) also decodes byte-exact: N independent nz_cd streams, each with its own FRESH ring (unlike the single-container case, where one ring is shared across the whole archive — confirmed empirically: each parallel-encoder thread owns its own nz_cd instance), sized per-slice via the same round(slice_size/0x10000)·0x10000 formula; each stream's type-0 chunks are already-delimited raw DecLZ blocks (no concatenation needed, unlike lzpf), decoded one at a time into the slice and verified against its own per-slice checksum before being tiled into the assembled output. Verified byte-exact on parallel containers up to 20 MB (text, random, and mixed text+random content). -cD (nz_lzhds = "lzhd strong") shares this ENTIRE front end (same token assembler, same column decoding, same LZ match/rep-cache model — even the same FUN_08099050-equivalent recon loop, dispatched via a vtable slot) and differs only in the literal coder: a per-context MTF rank table (256 contexts × 64 bytes: 32-slot rank table + 32-byte presence bitmap, seeded to the identity permutation with bitmap bits 0–31 set, not all-zero) plus a 4-stage adaptive linear predictor (pred = (w3·h3+w2·h2+w0·h0+0x200+w1·h1)>>10, sign-updated toward the residual), fed by an MSB-first bit-reader + two-level Exp-Golomb integer decoder over a dedicated -cD-only bitstream (one raw length byte + that many raw bytes, sitting between the token assembler's extra-bits blob and the literal column in the chunk layout — GDB-confirmed, not documented anywhere in the archive format itself). Ported in nz_lzhds.{h,cpp} and wired into both the single-container and parallel-container -cD paths in TryDecodeLegacyLzhd. Remaining bridge cases: rare CM/BWT sub-chunks (shared with -cd, e.g. stereo audio). The old DecLZ (nz_lzhd.cpp, nzdec_v0) is the wrong format for linux32 -cd/-cD.
optimum decoder (-co/-cO) ✅ both native single-container AND parallel-container LZ / ⚠️ BWT still bridges Three RE sessions of GDB tracing against the real linux32/nz binary found the real -co/-cO decode core is not DecLZ (the community-reference codec that name was borrowed from is never called) — it is a distinct LZMA-shaped LZ77 engine with 4 repeat-offset slots (rep0–3), a unary/Elias-gamma-style length/distance code, and an adaptive literal mixer sharing its low-level binary range coder with nz_cm.cpp. -co (FUN_0809e600, the "compact" nz_optimum1 variant, 4-context literal mixer) ported in nz_optimum_lz.{h,cpp}; a real port bug (the literal mixer's context-C seed sign-extended a byte the real binary zero-extends before shifting) was found and fixed via GDB divergence-tracing against a real failing archive. Also ports NzTextTransformRle (tt_flags & 0x20) and NzTextTransformInsertLf (tt_flags & 0x02). Native for -co single-container AND parallel-container (flag 0x0f, >~8 MB — reuses -cf/-cd's already-solved chunk-record format verbatim) archives with decr_param==1 (LZ/CM) blocks; still bridges on decr_param==0 (BWT). -cO (FUN_080a5d90, the "large" nz_optimum2 variant) shares the same backbone but a materially richer 8-context literal mixer plus a rolling 3-byte-hash LZP-style secondary predictor with no -co analog — ported in nz_optimum2_lz.{h,cpp}, reusing nz_cm.cpp's own kLzModelLNext/CM_Input_Bit primitives and -co's own dumped tables directly (GDB-confirmed byte-identical, not re-embedded) rather than re-deriving them. Wired into TryDecodeLegacyOptimum for method_p0==6, checksum-gated identically to -co. Found and fixed 3 real port bugs via GDB ground-truth (a units-to-bytes conversion in the dispatch-bit's second APM stage, a missing counter-based addressing scheme in rep-slot selection, and a copy-paste wrong base address in the length decoder's extra-bits loop) plus a 4th, subtler divergence that only manifested once mixer/weight state moved past the cold-start case (needed a longer, real-world-shaped stress fixture to even surface). Native for -cO single-container AND parallel-container (flag 0x0f) archives with decr_param==1 blocks — the parallel-container branch is shared verbatim with -co (the decoder type is selected once via a small closure since it's fixed per-archive, not per-stream), no new RE needed. decr_param==0 (BWT) still bridges for both -co and -cO.
cm decoder (-cc) ✅ native byte-exact Native CM decoder (NZ_CM.cpp, 1100 LOC). The documented "byte-26" divergence is FIXED (2026-06-08): it was a one-line port error in CM_Input_Bit — factors0_err used truncating factors[0] / 16 instead of the reference arithmetic shift factors[0] >> 4 (differs for negative values, flipping the factor[7] zeroing condition). Found by a per-bit next_probability diff vs a compiled reference oracle. With the full post-filter pipeline ported — param2 RLE (NzBwtRleDecodeU32), param1 AddBytes delta filter (NzAddBytesFilter), tt08 dictionary, tt16 number-transform (NzTextTransformNumber) — plus multi-chunk decoding (CM state persists across consecutive type-0 chunks), stored blocks (param6==0 = raw payload, for incompressible data), and NzTextTransformInsertLf (tt_flags & 0x02, shared with -co), -cc decodes byte-exact natively (no bridge) on source, numbers, dates, IPs, prose, markdown, random/large multi-chunk: 13/13 on a comprehensive sweep, 9/10 on native_only_v2 (the one miss is the deferred stereo audio-CM variant). NzAddBytesFilter's BitReader had a latent word-boundary truncation bug (rounded its side-stream length down to a multiple of 4 bytes instead of using the exact byte count, matching the reference exactly except for this one truncation) that a real-world corpus sweep found — it silently failed (clean decline, never wrong bytes) on any real file whose param1 side-stream length wasn't a multiple of 4, undetected by the synthetic test corpus; fixed, and the fix transparently benefits -co/-cO too since they call the same function. Cross-chunk continuation across a stored block was also missing: when a param6==0 stored (raw) chunk sits between two decr_param==0 CM chunks, the CM model must observe those stored bytes (model-update only, via NzCmFeedByte) to stay in sync with the true output position for the next CM chunk — found on a real file (integrty.doc), fixed, -cc real-corpus 31→32/56.
Encode for all methods ⚠️ functional Native BWT/store/literal writers; the legacy compression bridge was disabled in commit 049d041 (IsInternalLegacyCompressionBridgeCompressor returns false unconditionally). Codecs the native encoder cannot handle now produce an explicit error. All 8 methods byte-exact round-trip via the native encoder for low-entropy inputs.

Fixture-based benchmark (coverage_matrix.sh)

The coverage_matrix.sh test uses a deterministic AES-CTR-of-zeros fixture (low entropy). On that fixture:

native_strict_percent = 100% (8/8) — no bridge, no compat, no original binary subprocess.

Low-entropy input rarely triggers the prefilter+arith block mode, so the fixture passes fully native even though that path is not yet ported. Real-world compressible data (text, source code, binaries) will hit that path and fall back to the extract bridge for -cf/-cF.

Important caveat: native_strict_percent measures "no [bridge] / [compat] log line in stdout". It does NOT measure "the C++ code path actually ran the native decode vs silently called the legacy binary via FindLegacyBackend". A fixture that triggers the prefilter+arith block mode shows the bridge being invoked at runtime but still reports native_strict_percent=100 because the bridge produces byte-exact output. To verify the code path is truly native, use tests/native_only_v2.sh (NZ_NO_BRIDGE=1) — see the README's measured table, which is the honest number.

Real-corpus standing (60 files, 2026-09-01, all measured on the same corpus)

codec pass codec pass
-cn 60/60 -cd 60/60
-cf 60/60 -cD 60/60
-cF 60/60 -cO 59/60
-co 58/60 -cc 56/60

473/480 overall. All five codecs that are perfect here — -cn, -cf, -cF, -cd, -cD — cover the whole 0x2b family plus store. Every remaining failure is in -co/-cO/-cc.

(2026-09-01: -cD stage-index fix c831c31, -cf/-cF exe-filter fix f885e1c, prefilter hash-backfill fix eb66e8c, prefilter sub-chunk cd5e3f3, and the two cross-chunk state fixes 7a4d4c5 that took -cd 58 → 60 and -cD 52 → 60.)

Measuring every codec on ONE corpus is what surfaced the -cD deficit (then 34/60); comparing numbers across differently-generated corpora had been hiding it. Four files (Moly, summer.php, M05.AMF, RESOURCE.001) fail across most codecs and are the shared hard set.

Open work (as of 2026-09-01, after 7f07b6d)

Synthetic suite: 84/88 (native_only_v2.sh, 11 fixtures × 8 methods). Real corpus (60 files, all codecs on the same corpus): -cn 60 · -cf 60 · -cF 60 · -cd 60 · -cD 60 · -cO 59 · -co 58 · -cc 56 — 473/480.

No unported post-filter or chunk kind remains, and the whole 0x2b family plus store is clean on the real corpus. Every remaining failure is in -co/-cO/-cc.

The 11 remaining failures are THREE defects, not eleven

Classified by tracing each one (NZOPT_TRACE_TDO=1) rather than by filename — the previous revision of this page grouped them wrongly, calling six of them a single "wrong-bytes class" and the audio gap "the last 2 synthetic failures".

defect corpus failures synthetic failures
1. decr_param == 2 audio 5 4
2. -cc BWT wrong bytes 1 0
3. -cO LZ model divergence 1 0

1. decr_param == 2 audio blocks — 9 of the 11, by far the biggest item

-co M05.AMF, -co RESOURCE.001, -cc Moly, -cc M05.AMF, -cc RESOURCE.001, plus both audio-bearing synthetic fixtures on both -co and -cc. Two distinct symptoms, same defect: the audio decoder either returns 0 outright (audio Decode(...) -> 0) or completes and fails the checksum.

GDB ground truth (2026-09-01, 926a2a0) — the notes' framing of this was wrong. This is not a separate algorithm family needing a fresh port. Watchpointing the residual array the real decoder hands to its reconstruction call (FUN_080a50c0, 3rd argument), then watching one element of it to recover which stages write it, gives:

codec real stage sequence
-cO FUN_0809bbf0 → FUN_08095d90 → FUN_08096160
-co FUN_0809bbf0 → FUN_080958d0 → FUN_08095d90 → FUN_08096e20
-cc same shape as -co

The audio path is assembled from the same primitives as the lzpf prefilter, three of which are already byte-exact in the tree:

  • FUN_0809bbf0 = nzr::lzpf::DecodeResidualsStereo — whose own in-tree comment calls it speculative and "never the real path". That is true for lzpf and false for audio: this is its real caller.
  • FUN_08095d90 = nzr::lzpf::LpcPredictor / PrefilterPlane
  • FUN_08096e20 = nzr::lzpf::LmsObject (the LMS)

Two are not ported: FUN_080958d0 (an extra predictor stage that only -co/-cc run) and FUN_08096160 (-cO's inter-channel stage, where -co/-cc use the LMS instead). That one difference in the stage list is the whole reason the AudioPred transcription in nz_audio.cpp agrees with the binary under -cO and nowhere else — it is a different implementation that happens to coincide for -cO's stage combination.

Measured facts worth not re-deriving:

  • The residual array handed to the reconstruction is byte-identical across -co, -cO and -cc (32000 int32, md5 7d4ca807c2b0fdcc6f57ac92291a82f9).
  • This port reproduces it byte-exactly for -cO (0 diffs) and gets it wrong for -co (from element 0) and -cc (from element 2). So the reconstruction stage is fine; the defect is entirely in residual production.
  • The real decoder writes resid[0] exactly once — its predictor stages never touch the first element.

Dump this port's residuals with NZOPT_DUMP_AUDRESID=<path> and diff against a fresh capture. The three repro archives are in ~/.cache/nzre_aud/ (a_co.nz, a_cO.nz, a_cc.nz + stereo_lms.wav); the legacy binary decodes all three byte-exactly.

Progress, 2026-09-01 (f54131b) — one of the two differences fixed

The flag byte GDB reads at FUN_080a5330's entry is -cO 0x03, -cc 0x0f, -co 0x13, and bit 4 selects the inter-channel stage: clear → FUN_08096160 (two 4-bit shifts biased +0x10), set → FUN_08096e20 (two 3-bit shifts biased +7). -co is the only one of the three with bit 4 set, and the port had implemented the clear branch only — it always read 4+4 bits, so -co consumed two bits too many and desynchronised every side-channel field after it. Fixed by wiring nzr::lzpf::LmsObject + ApplyLmsInterChannel (already byte-exact in-tree) as the alternative branch, plus the both-objects reset the decompile does when the gating bit is clear. -co's residual error dropped 127427 → 103969 of 128000 bytes; no file flips yet and nothing regressed.

Also corrected: there is no missing FUN_080958d0. An earlier note here named it as an unported extra predictor stage, from a PC seen writing the residual array. Reading the call that led there shows call 0x08095d90 — 0x08095ca5 is a branch inside FUN_08095d90 (its order<9 / order>=9 split), below its own entry. The "last push %ebp before the PC" heuristic that produced the wrong name is unreliable here: several functions in this region open with sub $N,%esp instead.

The remaining difference — the residual front end

-co's residuals are still wrong from element 0, which is upstream of every predictor and of the inter-channel stage. The decompile's front end is

FUN_080c0630(ctx, 0xc, 3);  FUN_080a4ea0(...) x2;  FUN_0809bbf0(dst, n, counts, br)

— the generic arith buffer decoder (DecodeArithBuffer, already in-tree) run twice for the two per-channel count arrays, then the stereo residual decoder. nz_audio.cpp instead uses the community reference's bespoke AudioBitcountDecoder + DecodeInt32Array. Those two agree for -cO and disagree for -co/-cc; the binary has exactly one implementation, so the port's is the one that has to go. Replacing that front end is the next change, and it must keep -cO byte-exact.

Note -cc has bit 4 clear, i.e. the same inter-channel branch as the working -cO, so its failure was never the inter-channel stage — it is this front end (its residuals diverge at element 2). The flag bytes differ in bits 2-3 (0x0f vs 0x03), which FUN_080a5330 itself never tests, so those bits act inside the callees.

Refuted, do not retry: sweeping the third predictor pair's Initialize order over 13 values; flipping RunSmall's delta sign to match UpdateBig (breaks the working -cO); reading the small header the reconstruction receives (01 01 02 02 2c 00 00 00 ..., where [4..7] is header_bytes = 44 and [2]/[3] drive FUN_080a50c0's branch) as the six predictor-enable flags — no bit offset in −120..+120 reproduces it and forcing the pattern does not match; and re-routing the payload straight into DecodePrefilterStream (best of 64 offsets × orders × nstages × mono/stereo still left 63725 of 64044 bytes wrong).

2. -cc on audio/01 Track 01.m4a — BWT wrong bytes

Three decr_param == 0 blocks, no audio block at all. The sequence completes with ok=1 at the full 2920622 bytes and the entry checksum rejects it. This is the only pure wrong-bytes case left, and the oracle situation is good: -cf, -cF, -cd and -cD all decode this file byte-exactly.

3. -cO on doc/Fonts Poster-color.dp — one mis-decoded literal bit

Localised precisely, so this is a warm start rather than a fresh investigation:

  • Block 1 (19219 bytes) is byte-exact. Block 2 is 122944 bytes, decoded as four 0x8000 chunks.
  • Chunks 1-3 (ring 19219..117522, 98304 bytes) are completely correct: 91439 literals, zero wrong.
  • The ring then wraps (EnsureHeadroom, cursor -> 0). Chunk 4 stays correct for 2922 more bytes and 59 further matches, then emits one wrong literal byte at chunk-4 ring position 2923 (file offset 122018): 0x80 where 0x40 belongs — a single bit in the literal bit-tree.
  • Everything after that is downstream damage. The eventual visible error is FAIL@distance pos=10568 acc=385363 capacity=131072, ~7.6k later: a bogus slot 18 (the slot ladder covers distances (2^s, 2^(s+1)], so with a 2^17 window the maximum legal slot is 16).

So this is a latent adaptive-model state divergence that only flipped a decision after ~101k correct bytes, not a structural bug at the wrap. The wrap is still the first thing to re-examine (EnsureHeadroom memsets [cursor, cap+256) on the first scroll and then copies the last 256 bytes to base-256; if the real code copies BEFORE zeroing, the 256-byte backward guard differs).

Measuring gotcha, worth reading before touching this: the traced position is a RING position and the ring is NOT contiguous with file offsets across a wrap, nor across a block that has a param2 layer (the ring holds the pre-post-filter output). Validating match distances against file offsets naively produces false positives — it reported a bogus "first invalid match at pos=608" that was in fact perfectly correct once mapped properly.

4. Latent, unverified

  • 0xd / 0xe chunk nibbles are reachable but never observed in a 71-file plus 90-file sweep; the flags & 2 reset path for them is wired but untested.
  • decr_param == 3 never occurred in 654 blocks across 357 archives; the port parses its header shape and declines.
  • tt bit 0x20 is wired for -cc but no -cc archive in the corpus sets it, so it is unverified there.
  • The -cD pure-literal model reset in 7a4d4c5 is scoped to the size_field == 0 flavour of pure-literal chunk. The v2 == 0 flavour demonstrably must NOT reset, and a prefilter sub-chunk must not either, but the underlying mechanism in the binary was inferred from behaviour rather than read out of the decompile — if a future file disagrees, that scope is the first thing to re-measure.
  • A stored LZ block (decr_param == 1 with param6 == 0, fixed in 7f07b6d) has no synthetic reproduction. Eight attempts to construct one — pure high-entropy at several sizes, entropy+text mixes, a restricted-nibble alphabet, compressible-header + entropy layouts — all made the encoder emit param6 == 1 for every LZ block and param6 == 0 only for BWT blocks. The real-corpus sweep is the only guard; NZOPT_TRACE_TDO now prints "stored LZ block" when the shape appears.

Closed since the last revision (both were misdiagnosed here)

The previous two entries on this list were -cD order-32 predictor and -cd residuals. Both are closed by 7a4d4c5, and neither had the cause this page attributed to it:

  • The order-32 predictor was not the problem. This page said to diff the in-tree LpcBigPredictor against the validated ord32.h — that diff is empty, the two are byte-identical. The lead was plausible only because -cd configures order 8 (so LpcBigPredictor never runs there) and -cD order 32; but the failures had nothing to do with the predictor.
  • The -cd residuals (46 and 75 bytes) were not an LMS or state-lifetime detail either. They were the same stale ring cursor after a prefilter sub-chunk that -cD was hitting.

Method notes worth keeping

  • Measure every codec on ONE corpus before choosing work. Doing that is what revealed -cD sitting 22 points below its sibling; comparing numbers across differently-generated corpora had hidden it.
  • A file failing under many codecs means a shared code path, not inherent difficulty. The four-file "hard set" turned out to be one defect in -cf/-cF and a different single defect in -cd/-cD.
  • "Right framing, wrong content" does not distinguish a missing core from a misconfigured one. That inference produced two wrong labels in the notes and cost several sessions.
  • Closing a coverage gap routinely exposes latent UB in newly-reached code — re-run ASAN after every gap fix, not only after touching a file you sanitised before. It has happened four times.
  • Fuzz the decoder on corrupt input. Byte-exactness testing cannot find memory-safety bugs; single-byte corruption under ASAN found an out-of-bounds heap write.
  • Verify a stored diagnosis before acting on it — the cheapest step is often a diff. The top item on this list for a whole session was "port the order-32 predictor, start by diffing against the validated ord32.h". Running that diff took one command and returned empty: the file was already in the tree. A recorded next-step can be stale or simply wrong; re-establish it before spending a session on it.
  • Group failures by WHERE they diverge, not by which file they are. Six unrelated-looking -cD failures (an m4a, a .adf, an .EXE, a module, a PHP source, a bitmap) all diverged at or within a few bytes of a 32768-byte chunk boundary, and in every case the preceding chunk was one that BYPASSES the codec's model. That single measurement — cmp offset vs. the chunk table — reframed the work from "port a predictor" to "find the dropped cross-chunk state", and it is a two-minute measurement.
  • Out-of-distribution output bytes name the guilty stage. In smooth audio-like data, emitting 01 03 02 06 where 95 94 91 95 belongs is not a wrong match copy (that would emit plausible bytes from elsewhere) — it is a predictor fed a wrong base. Read the values, not just the offsets.
  • Trace the primitive's inputs, not only its outputs. The -cD MTF bug was unreadable from output bytes alone, and obvious the moment the trace printed ctx=3a rank=1d -> sym=3a: the expected symbol was the rank code, which is what a freshly initialised table returns. One extra field in a printf.
  • "State persists across chunks" and "state resets every chunk" can both be wrong. For the nz_lzhds MTF table the rule is conditional — persist across token chunks and prefilter sub-chunks, reset across a full stored chunk. Test the reset scope in BOTH directions: an over-broad reset and an under-broad one both fail, on different files.
  • Any codec that is byte-exact on a file is an oracle for every codec that is not. With -cf/-cF/ -cd/-cD now clean on all 60 corpus files, every -co/-cO/-cc failure has four independent byte-exact references for the same plaintext.
  • Classify every failure before choosing which to fix. Reading one trace line per failing file (decr_param, and whether the sequence completed) turned "6 files in a wrong-bytes class + 2 synthetic audio failures" into "one audio defect accounting for 9 of 11 failures, plus two singletons". That changes what to work on next, and it cost about ten minutes.
  • A reference implementation reading an uninitialised field is not a specification. NZ.cpp takes an LZ block's output size from size18, which is only assigned under if (param6) — so at param6 == 0 it is UB. When the reference is silent or undefined, make the sizes add up against a real file instead.
  • When a fix has no synthetic reproduction, say so in the commit. Eight constructed shapes failed to make the encoder emit a stored LZ block; recording that stops the next session from repeating the attempts and stops anyone from believing the suite guards it.

Clone this wiki locally