Component Status

Component status

Detailed per-codec native-decode status. See the README for the measured summary table and quick start; see Changelog for the day-by-day history of how each status below was reached.

Known gaps where the bridge is invoked at runtime

(The bridge produces byte-exact output but the C++ does not decode these paths natively.)

• lzpf stereo prefilter (-cf AND -cF) is now native end-to-end (see the component table below). The remaining -cF bridge case is the non-prefilter LZ77 variant-B path on some inputs.
• lzhd (-cd/-cD): the coroutine token-LZ is native incl. multi-chunk text, large multi-stream files, and parallel containers (see the component table below). The LZ window is a single per-archive ring (single-container) or a fresh per-stream ring (parallel container) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB; modular helpers so non-power-of-two sizes work) — confirmed by GDB on FUN_08099050 (obj+0x978): 1/3/19/46 × 64 KB for text50/source.cpp/big_code/repeat_3M. The ring is sized to hold the whole compact recon, so its cursor advances monotonically and never wraps for real archives; it persists across the archive's 1 MB output streams. -cD (nz_lzhds) shares this entire front end and adds its own literal model — a per-context MTF rank table + adaptive order-N linear predictor, fed by an MSB-first bit reader + Exp-Golomb integer decoder (nz_lzhds.{h,cpp}) — now ported and native. The decoder still self-verifies against the stored per-file checksum and bridges on any mismatch (defense in depth). Remaining bridge cases: rare CM/BWT sub-chunks (shared by -cd/-cD).
• optimum1 (-co) single-container decr_param==1 (LZ/CM) blocks are now native (see the component table below); an older theory that large -co/-cO used an unreversed "virtual-stream" DecLZ framing was disproven by GDB tracing against the real binary — the actual blocker was that DecLZ (a port of the community reference decoder) is simply never called by the real binary at all. The parallel multi-stream container (flag 0x0f, >~8 MB) is also native now for BOTH -co and -cO, reusing the same chunk-record format as -cf/-cd's parallel containers. optimum2 (-cO) single-container decr_param==1 blocks are native too now (its richer 8-context literal mixer + LZP secondary predictor ported into nz_optimum2_lz.{h,cpp}). Still bridging for both -co and -cO: decr_param==0 (BWT) blocks, and a newly-found single-container "chain" mode (multiple stream_tag segments in one archive — see the roadmap below).
• cm stereo audio variant deferred; dece (exe-filter) post-filter not yet ported (rare). param1 (AddBytes) is now ported and working for -cc/-co/-cO — a real-world corpus sweep found and fixed a latent word-boundary truncation bug in its BitReader that had silently made it fail on real files whose side-stream length wasn't a multiple of 4 bytes.

Roadmap to 100% native decode

1. ✅ Done: NZ_NO_BRIDGE=1 flag — when set, FindLegacyBackend* return empty and a missing native path is a hard error (no silent $PATH//usr/bin/nz fallback). This is what native_only_v2.sh uses to measure honestly.
2. ✅ Done: the -cd cross-chunk / cross-stream LZ window. It is a single per-archive ring of size round(total_output / 0x10000) · 0x10000 (min 64 KB; GDB-confirmed on FUN_08099050 obj+0x978) that persists across the archive's 1 MB output streams (NzCdDecodeStream threads the ring position and the file-absolute output offset). The ring is sized to hold the whole compact recon, so it never wraps for real archives. This made multi-chunk text and large multi-stream files (a 7-chunk source file; 2–3.5 MB code/text/base64 incl. heavily-repetitive input) byte-exact native, with a checksum self-verify → bridge fallback as defense in depth.
3. ✅ Done (-co/-cO, single-container AND parallel-container): ported the real linux32 -co/-cO LZ/CM cores (nz_optimum_lz.{h,cpp}, nz_optimum2_lz.{h,cpp}), LZMA-shaped LZ77 engines with 4 repeat-offset slots — not DecLZ, which is never called by the real binary. Also ✅ done: the tt_flags & 0x02 (InsertLF) text-transform bit for both -co and -cc; and param1 (AddBytesFilter) for -cc/-co/-cO. Remaining optimum work:
   • decr_param==0 (BWT) for both -co and -cO.
   • A newly-found single-container "chain" mode (multiple stream_tag segments in one archive). A real small .doc file surfaced this: TryDecodeLegacyOptimum currently parses exactly one top-level stream_tag per single-container archive, but a real file's archive can have a second segment covering the remaining output. An earlier session's negative "chain mode doesn't exist for -co" conclusion was apparently based on insufficient fixtures (large synthetic files only, never a small real multi-block document).
4. Port the remaining rare post-filter (dece exe-filter) and the CM stereo-audio variant.
5. Extend real stereo/multichannel audio decode to -cd/-cD/-cc/-co/-cO — a real-world corpus sweep found these codecs have near-zero coverage on real audio content (unlike -cf/-cF, whose stereo prefilter was fixed to 9/9 on the same corpus). These codecs don't have a dedicated prefilter like -cf/-cF — the gap is presumably in their own entropy/CM paths or the shared decr_param==0/BWT path.
6. Fix a separate bug in param2's NzBwtRleDecodeU32 RLE expander, found (but not yet root-caused) via the same real-world corpus sweep that found the param1 bug above — a real file (integrty.doc) still declines under -cc even after that fix.

Component table

Component	Cloned?	Notes
CLI structure (l/t/x/a/s, all switches)	✅ 100%	Full parser
Archive format (header, entry table, stream families 0x2b/0x3b/0x4b)	✅ 100%
Store (-cn)	✅ 100%	Single-block stores are a contiguous tail copy. Files large enough to split (> ~0x30000) are stored as multiple blocks — [varint (len<<4)|0][raw] with a per-block checksum trailer between blocks — reassembled by TryAssembleStoredBlocks (-cn 9/10→10/10; the prior miss was a 215 KB source file stored as 196608 + 18566).
lzpf decode LZ77+arith path (-cf/-cF)	✅ native (10/10 measured)	LZ77+arith (variants A 13-bit hash, B 24-bit hash) + literal + raw-bytecode block modes native. Native byte-exact on most real inputs under NZ_NO_BRIDGE=1. Large single-container multi-stream -cf/-cF (1–8 MB) decode byte-exact: the output splits into ~1 MB compressor streams chained [stream_tag][data] with an inter-stream whole-output checksum record (tag 0x45/0x47/0x26); dict/hash state persists across all streams. The parallel multi-stream container (header flag 0x0f, >~8 MB, multi-threaded encoder) also decodes byte-exact: N independent nz_lzpf streams, each a group of chunk records (type-1 slice size, type-10 u32 output offset, type-5/7/6 slice checksum, one or more type-0 compressed chunks concatenated into the stream), each decoded and placed at its offset and required to tile the whole output, every byte slice-checksum-verified. The sliding-window dict capacity is a multiple of 64 KiB the encoder picks by its threading (one of (p1+1)·64 KiB, or total rounded down/up to 64 KiB / 128 KiB; GDB-confirmed on FUN_080b6bb0 obj+4), so the decoder tries each candidate and adopts the first whose decode matches the slice/whole-output checksum (an unmodelled window wrap is rejected, never emitted). Mixed-binary content (real-world tars with both text and incompressible/random data) also decodes byte-exact: literal blocks now backfill hash_table after every literal run (legacy FUN_080b6d90/FUN_080b6cf0 — sparse, every 101 bytes, or dense, every byte, selected by a bit in the block header; variant B additionally zeroes its byte_buffer_8k side-table per FUN_080b6c20). Without this the table's untouched init value (3) leaked into later f6/f8/medium-match copies whenever a match referenced a hash bucket only ever touched during a literal run — byte-correct through many blocks by chance, then producing sparse single-byte corruption the moment a genuinely-collided bucket got referenced. Root-caused by GDB-diffing the decoder's hash_table against ground truth at the exact block boundary where output first diverged. The "8/8 100%" figure is from the low-entropy coverage_matrix.sh fixture only and does NOT reflect real data — see the measured table in the README. Variant B hash init=3 fix (commit 049d041) closed the multi-file silent corruption.
lzpf prefilter+arith path (-cf)	✅ mono + stereo native	FUN_080a5330 + LPC filter (FUN_08095d90) ported; mono AND stereo audio byte-exact. The stereo path was closed end-to-end (stereo_lms_cf.nz → stereo_lms.wav, both blocks byte-exact, -cf native_only_v2 9/10→10/10). The full FUN_080a5330 stereo flow is now reproduced in DecodePFBlock: residuals are PLANAR (ch1=[0,per_chan), ch2=[per_chan,2·per_chan)) decoded by two per-channel DecodeArithBuffer calls (each reads its own Huffman header — a single n_elems call mis-locates the side stream by ~1.4 KB); the predictor-init reads a leading bit G (iStack_50078, gates the inter-channel LMS) then per channel [active(1), order(3 if active)] (order = bits+8); per-channel LPC with two persistent predictor states (pred/pred2, threaded across blocks); the verified inter-channel LMS FUN_08096e20 (ApplyLmsInterChannel, regression test TestLmsInterChannel); and FUN_080a50c0 reconstruction (ReconstructStereoSamples: per-channel delta-integrate + L/R (channels==1) or mid/side (channels==2) interleave). -cF (lzpf B / nz_lzpf_large) stereo is also native: it uses the SAME FUN_080a4ea0 arith as -cf (not a vtable path), differing only in the LPC filter order — FUN_08095d90 obj+0x1c08 is 8-tap for variant B vs 4-tap for variant A (GDB-confirmed), so LpcPredictor is now N-tap (8-tap adapts every sample, 4-tap every other). -cF native_only_v2 8/10→9/10. A real file-format-sample sweep later found the single-fixture validation above had missed real bit-depth/channel variation (5/9 on a 9-file real stereo/multichannel WAV corpus spanning 2/6 channels, 8/16/24-bit): fixed 4 real bugs — tail-remainder bytes silently dropped instead of copied verbatim; the generic (non-int16-fast-path) reconstruction always did plain L/R interleave instead of mid/side for channels==2; the LMS per-object adaptation-shift bits were never read/applied; and LPC/LMS predictor state wasn't reset across a literal/LZ77 block sitting between prefilter blocks (GDB-confirmed on the real binary: state is exactly zero after every non-prefilter block, carried forward after every prefilter block, 16-for-16). Real-corpus audio sweep now 9/9 for both -cf and -cF.
lzhd decoder (-cd/-cD)	✅ LZ (both -cd raw-literal and -cD MTF+predictor literal models) + block-RLE + raw-store + pure-literal + exe + text-pipeline (param14/line-RLE/CRLF/word-dict) + multi-chunk text + large multi-stream files + parallel containers native; only rare CM/BWT sub-chunks bridge	The real linux32 -cd is a coroutine token-LZ (NOT the reference DecLZ). Fully ported in nz_cd_tokens.{h,cpp} and validated byte-exact against the binary: token assembler (FUN_080aa070), reconstruction with trailing-literal flush (FUN_08099050), per-column RLE run-expander (FUN_080acb90, thr=1 for the LEN column / 0 for LIT·OFF), param14 text transform (FUN_080a0ff0), the bounded-varint header (FUN_080b1dc0), and the integrated chunk/stream/block decoders (NzCdDecodeLzChunk/NzCdDecodeStream/NzCdDecodeBlock). Columns/literals reuse DecodeArithBuffer. The reconstruction runs over a single per-archive ring (FUN_08099050, obj+0x978) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB) — GDB-confirmed across text50/source.cpp/big_code/repeat_3M = 1/3/19/46 × 64 KB. The size need not be a power of two, so the ring uses modular helpers (not a bitmask). The ring is sized to hold the whole compact recon, so its cursor (obj+0x980) advances monotonically and never wraps for real archives (the reset/wrap path is a safety fallback). Large files split their output into 1 MB streams; the ring is allocated once and persists across streams (NzCdDecodeStream threads ring_pos and the file-absolute output offset), so cross-stream matches resolve correctly. Wired into the extract dispatcher and BYTE-EXACT end-to-end under NZ_NO_BRIDGE=1 for: token-LZ (recon == file), block-RLE (flag &2, post-recon run-length re-expansion of collapsed zero-runs), raw-store (per-column b0&1==0 and flag-&1-clear literals = verbatim bytes), pure-literal (no LZ tokens — the whole window is one literal stream; generator picks this when size_field==0 or v2==0), exe (flag &4: a BCJ-style x86 E8/E9 call/jmp address un-transform, NzCdExeUnfilter), and the text pipeline (flag &8, FUN_080a3c90): param14 (NzCdParam14), line-RLE (FUN_080a2f20), CRLF EOL (FUN_080a19b0), and the word-dictionary transform (FUN_080a0a00 = the reference TransformText_1_Dictionary + dict/char-trait tables, in nz_cd_texttransform_dict.cpp). Verified on text (map.txt), binary (image.cat), an ELF (elf.bin, flags=3 block-RLE), an EXE (play.exe, flags=5), atoll (multi-chunk &8), word-dictionary text, multi-chunk text (text_50k/repeat_100k/CRLF), a 7-chunk source file (single-stream 192 KB ring), large multi-stream files (2–3.5 MB code/text/base64, incl. heavily-repetitive input), and a varied real corpus spanning images/audio/video/executables. As defense in depth TryDecodeLegacyLzhd self-verifies the decoded output against the archive's stored per-file checksum and returns false on mismatch, so any unforeseen edge falls through to a byte-exact bridge decode (no silent corruption, and NZ_NO_BRIDGE native-only is a provable correctness signal). The parallel multi-stream container (header flag 0x0f, >~8 MB, mirroring the -cf/-cF parallel format) also decodes byte-exact: N independent nz_cd streams, each with its own FRESH ring (unlike the single-container case, where one ring is shared across the whole archive — confirmed empirically: each parallel-encoder thread owns its own nz_cd instance), sized per-slice via the same round(slice_size/0x10000)·0x10000 formula; each stream's type-0 chunks are already-delimited raw DecLZ blocks (no concatenation needed, unlike lzpf), decoded one at a time into the slice and verified against its own per-slice checksum before being tiled into the assembled output. Verified byte-exact on parallel containers up to 20 MB (text, random, and mixed text+random content). -cD (nz_lzhds = "lzhd strong") shares this ENTIRE front end (same token assembler, same column decoding, same LZ match/rep-cache model — even the same FUN_08099050-equivalent recon loop, dispatched via a vtable slot) and differs only in the literal coder: a per-context MTF rank table (256 contexts × 64 bytes: 32-slot rank table + 32-byte presence bitmap, seeded to the identity permutation with bitmap bits 0–31 set, not all-zero) plus a 4-stage adaptive linear predictor (pred = (w3·h3+w2·h2+w0·h0+0x200+w1·h1)>>10, sign-updated toward the residual), fed by an MSB-first bit-reader + two-level Exp-Golomb integer decoder over a dedicated -cD-only bitstream (one raw length byte + that many raw bytes, sitting between the token assembler's extra-bits blob and the literal column in the chunk layout — GDB-confirmed, not documented anywhere in the archive format itself). Ported in nz_lzhds.{h,cpp} and wired into both the single-container and parallel-container -cD paths in TryDecodeLegacyLzhd. Remaining bridge cases: rare CM/BWT sub-chunks (shared with -cd, e.g. stereo audio). The old DecLZ (nz_lzhd.cpp, nzdec_v0) is the wrong format for linux32 -cd/-cD.
optimum decoder (-co/-cO)	✅ both native single-container AND parallel-container LZ / ⚠️ BWT still bridges	Three RE sessions of GDB tracing against the real linux32/nz binary found the real -co/-cO decode core is not DecLZ (the community-reference codec that name was borrowed from is never called) — it is a distinct LZMA-shaped LZ77 engine with 4 repeat-offset slots (rep0–3), a unary/Elias-gamma-style length/distance code, and an adaptive literal mixer sharing its low-level binary range coder with nz_cm.cpp. -co (FUN_0809e600, the "compact" nz_optimum1 variant, 4-context literal mixer) ported in nz_optimum_lz.{h,cpp}; a real port bug (the literal mixer's context-C seed sign-extended a byte the real binary zero-extends before shifting) was found and fixed via GDB divergence-tracing against a real failing archive. Also ports NzTextTransformRle (tt_flags & 0x20) and NzTextTransformInsertLf (tt_flags & 0x02). Native for -co single-container AND parallel-container (flag 0x0f, >~8 MB — reuses -cf/-cd's already-solved chunk-record format verbatim) archives with decr_param==1 (LZ/CM) blocks; still bridges on decr_param==0 (BWT). -cO (FUN_080a5d90, the "large" nz_optimum2 variant) shares the same backbone but a materially richer 8-context literal mixer plus a rolling 3-byte-hash LZP-style secondary predictor with no -co analog — ported in nz_optimum2_lz.{h,cpp}, reusing nz_cm.cpp's own kLzModelLNext/CM_Input_Bit primitives and -co's own dumped tables directly (GDB-confirmed byte-identical, not re-embedded) rather than re-deriving them. Wired into TryDecodeLegacyOptimum for method_p0==6, checksum-gated identically to -co. Found and fixed 3 real port bugs via GDB ground-truth (a units-to-bytes conversion in the dispatch-bit's second APM stage, a missing counter-based addressing scheme in rep-slot selection, and a copy-paste wrong base address in the length decoder's extra-bits loop) plus a 4th, subtler divergence that only manifested once mixer/weight state moved past the cold-start case (needed a longer, real-world-shaped stress fixture to even surface). Native for -cO single-container AND parallel-container (flag 0x0f) archives with decr_param==1 blocks — the parallel-container branch is shared verbatim with -co (the decoder type is selected once via a small closure since it's fixed per-archive, not per-stream), no new RE needed. decr_param==0 (BWT) still bridges for both -co and -cO.
cm decoder (-cc)	✅ native byte-exact	Native CM decoder (NZ_CM.cpp, 1100 LOC). The documented "byte-26" divergence is FIXED (2026-06-08): it was a one-line port error in CM_Input_Bit — factors0_err used truncating factors[0] / 16 instead of the reference arithmetic shift factors[0] >> 4 (differs for negative values, flipping the factor[7] zeroing condition). Found by a per-bit next_probability diff vs a compiled reference oracle. With the full post-filter pipeline ported — param2 RLE (NzBwtRleDecodeU32), param1 AddBytes delta filter (NzAddBytesFilter), tt08 dictionary, tt16 number-transform (NzTextTransformNumber) — plus multi-chunk decoding (CM state persists across consecutive type-0 chunks), stored blocks (param6==0 = raw payload, for incompressible data), and NzTextTransformInsertLf (tt_flags & 0x02, shared with -co), -cc decodes byte-exact natively (no bridge) on source, numbers, dates, IPs, prose, markdown, random/large multi-chunk: 13/13 on a comprehensive sweep, 9/10 on native_only_v2 (the one miss is the deferred stereo audio-CM variant). NzAddBytesFilter's BitReader had a latent word-boundary truncation bug (rounded its side-stream length down to a multiple of 4 bytes instead of using the exact byte count, matching the reference exactly except for this one truncation) that a real-world corpus sweep found — it silently failed (clean decline, never wrong bytes) on any real file whose param1 side-stream length wasn't a multiple of 4, undetected by the synthetic test corpus; fixed, and the fix transparently benefits -co/-cO too since they call the same function.
Encode for all methods	⚠️ functional	Native BWT/store/literal writers; the legacy compression bridge was disabled in commit 049d041 (IsInternalLegacyCompressionBridgeCompressor returns false unconditionally). Codecs the native encoder cannot handle now produce an explicit error. All 8 methods byte-exact round-trip via the native encoder for low-entropy inputs.

Fixture-based benchmark (coverage_matrix.sh)

The coverage_matrix.sh test uses a deterministic AES-CTR-of-zeros fixture (low entropy). On that fixture:

native_strict_percent = 100% (8/8) — no bridge, no compat, no original binary subprocess.

Low-entropy input rarely triggers the prefilter+arith block mode, so the fixture passes fully native even though that path is not yet ported. Real-world compressible data (text, source code, binaries) will hit that path and fall back to the extract bridge for -cf/-cF.

Important caveat: native_strict_percent measures "no [bridge] / [compat] log line in stdout". It does NOT measure "the C++ code path actually ran the native decode vs silently called the legacy binary via FindLegacyBackend". A fixture that triggers the prefilter+arith block mode shows the bridge being invoked at runtime but still reports native_strict_percent=100 because the bridge produces byte-exact output. To verify the code path is truly native, use tests/native_only_v2.sh (NZ_NO_BRIDGE=1) — see the README's measured table, which is the honest number.