Skip to content

Component Status

Jump to bottom
YadeWira edited this page Jul 30, 2026 · 36 revisions

Component status

Detailed per-codec native-decode status. See the README for the measured summary table and quick start; see Changelog for the day-by-day history of how each status below was reached.

Known gaps where the bridge is invoked at runtime

(The bridge produces byte-exact output but the C++ does not decode these paths natively.)

  • lzpf stereo prefilter (-cf AND -cF) is now native end-to-end (see the component table below). The remaining -cF bridge case is the non-prefilter LZ77 variant-B path on some inputs.
  • lzhd (-cd/-cD): the coroutine token-LZ is native incl. multi-chunk text, large multi-stream files, and parallel containers (see the component table below). The LZ window is a single per-archive ring (single-container) or a fresh per-stream ring (parallel container) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB; modular helpers so non-power-of-two sizes work) — confirmed by GDB on FUN_08099050 (obj+0x978): 1/3/19/46 × 64 KB for text50/source.cpp/big_code/repeat_3M. The ring is sized to hold the whole compact recon, so its cursor advances monotonically and never wraps for real archives; it persists across the archive's 1 MB output streams. -cD (nz_lzhds) shares this entire front end and adds its own literal model — a per-context MTF rank table + adaptive order-N linear predictor, fed by an MSB-first bit reader + Exp-Golomb integer decoder (nz_lzhds.{h,cpp}) — now ported and native. The decoder still self-verifies against the stored per-file checksum and bridges on any mismatch (defense in depth). Remaining bridge cases: rare CM/BWT sub-chunks (shared by -cd/-cD).
  • optimum1 (-co) single-container decr_param==1 (LZ/CM) blocks are now native (see the component table below); an older theory that large -co/-cO used an unreversed "virtual-stream" DecLZ framing was disproven by GDB tracing against the real binary — the actual blocker was that DecLZ (a port of the community reference decoder) is simply never called by the real binary at all. The parallel multi-stream container (flag 0x0f, >~8 MB) is also native now for BOTH -co and -cO, reusing the same chunk-record format as -cf/-cd's parallel containers. optimum2 (-cO) single-container decr_param==1 blocks are native too now (its richer 8-context literal mixer + LZP secondary predictor ported into nz_optimum2_lz.{h,cpp}). Single-container archives with more than one stream_tag segment ("chain" mode — see the roadmap below) are also native now for both. Still bridging for both -co and -cO: decr_param==0 (BWT) blocks.
  • cm stereo audio variant deferred; dece (exe-filter) post-filter not yet ported (rare). param1 (AddBytes) is now ported and working for -cc/-co/-cO — a real-world corpus sweep found and fixed a latent word-boundary truncation bug in its BitReader that had silently made it fail on real files whose side-stream length wasn't a multiple of 4 bytes.

Roadmap to 100% native decode

  1. ✅ Done: NZ_NO_BRIDGE=1 flag — when set, FindLegacyBackend* return empty and a missing native path is a hard error (no silent $PATH//usr/bin/nz fallback). This is what native_only_v2.sh uses to measure honestly.
  2. ✅ Done: the -cd cross-chunk / cross-stream LZ window. It is a single per-archive ring of size round(total_output / 0x10000) · 0x10000 (min 64 KB; GDB-confirmed on FUN_08099050 obj+0x978) that persists across the archive's 1 MB output streams (NzCdDecodeStream threads the ring position and the file-absolute output offset). The ring is sized to hold the whole compact recon, so it never wraps for real archives. This made multi-chunk text and large multi-stream files (a 7-chunk source file; 2–3.5 MB code/text/base64 incl. heavily-repetitive input) byte-exact native, with a checksum self-verify → bridge fallback as defense in depth.
  3. ✅ Done (-co/-cO, single-container AND parallel-container): ported the real linux32 -co/-cO LZ/CM cores (nz_optimum_lz.{h,cpp}, nz_optimum2_lz.{h,cpp}), LZMA-shaped LZ77 engines with 4 repeat-offset slots — not DecLZ, which is never called by the real binary. Also ✅ done: the tt_flags & 0x02 (InsertLF) text-transform bit for both -co and -cc; and param1 (AddBytesFilter) for -cc/-co/-cO. Remaining optimum work:
    • decr_param==0 (BWT) for both -co and -cO — now the dominant real-world decline reason for both codecs (confirmed via a 56-file real-corpus sweep).
    • ✅ Done: single-container "chain" mode (multiple stream_tag segments in one archive, directly concatenated with no separator). A real 60 KB .doc file surfaced this — its first segment covered only 10229 of 60416 declared output bytes, and the very next byte was a second, independently valid stream_tag varint. An earlier session's negative "chain mode doesn't exist for -co" conclusion was based on insufficient fixtures (large synthetic files only, never a small real multi-block document). Fixed by looping over segments with one persistent decoder shared across the whole entry. Real-corpus sweep: -co 20→26/56, -cO 21→27/56.
  4. Port the remaining rare post-filter (dece exe-filter) and the CM stereo-audio variant.
  5. Extend real stereo/multichannel audio decode to -cd/-cD/-cc/-co/-cO — a real-world corpus sweep found these codecs have near-zero coverage on real audio content (unlike -cf/-cF, whose stereo prefilter was fixed to 9/9 on the same corpus). These codecs don't have a dedicated prefilter like -cf/-cF — the gap is presumably in their own entropy/CM paths or the shared decr_param==0/BWT path.
  6. ✅ Done: integrty.doc declining under -cc was initially misattributed to param2's NzBwtRleDecodeU32 RLE expander (the function that returned false), but tracing one level upstream showed param2 was correct — the CM chunk feeding it had already produced garbage output partway through. Root cause: when a decr_param==0 chain has a param6==0 STORED (raw) chunk sitting between two CM-coded chunks, the CM model's rolling byte-window/hash context must still observe those stored bytes to stay in sync with the true output position, even though they were never CM-arithmetic-coded — the port never did this. Confirmed by building the community reference decoder standalone and reproducing the identical corruption on the real archive's exact bytes, then confirming the fix (feed stored bytes through a model-update-only path, NzCmFeedByte) against that same reproduction. Real-corpus sweep: -cc 31→32/56.

Component table

Component Cloned? Notes
CLI structure (l/t/x/a/s, all switches) ✅ 100% Full parser
Archive format (header, entry table, stream families 0x2b/0x3b/0x4b) ✅ 100%
Store (-cn) ✅ 100% Single-block stores are a contiguous tail copy. Files large enough to split (> ~0x30000) are stored as multiple blocks — [varint (len<<4)|0][raw] with a per-block checksum trailer between blocks — reassembled by TryAssembleStoredBlocks (-cn 9/10→10/10; the prior miss was a 215 KB source file stored as 196608 + 18566).
lzpf decode LZ77+arith path (-cf/-cF) ✅ native (10/10 measured) LZ77+arith (variants A 13-bit hash, B 24-bit hash) + literal + raw-bytecode block modes native. Native byte-exact on most real inputs under NZ_NO_BRIDGE=1. Large single-container multi-stream -cf/-cF (1–8 MB) decode byte-exact: the output splits into ~1 MB compressor streams chained [stream_tag][data] with an inter-stream whole-output checksum record (tag 0x45/0x47/0x26); dict/hash state persists across all streams. The parallel multi-stream container (header flag 0x0f, >~8 MB, multi-threaded encoder) also decodes byte-exact: N independent nz_lzpf streams, each a group of chunk records (type-1 slice size, type-10 u32 output offset, type-5/7/6 slice checksum, one or more type-0 compressed chunks concatenated into the stream), each decoded and placed at its offset and required to tile the whole output, every byte slice-checksum-verified. The sliding-window dict capacity is a multiple of 64 KiB the encoder picks by its threading (one of (p1+1)·64 KiB, or total rounded down/up to 64 KiB / 128 KiB; GDB-confirmed on FUN_080b6bb0 obj+4), so the decoder tries each candidate and adopts the first whose decode matches the slice/whole-output checksum (an unmodelled window wrap is rejected, never emitted). Mixed-binary content (real-world tars with both text and incompressible/random data) also decodes byte-exact: literal blocks now backfill hash_table after every literal run (legacy FUN_080b6d90/FUN_080b6cf0 — sparse, every 101 bytes, or dense, every byte, selected by a bit in the block header; variant B additionally zeroes its byte_buffer_8k side-table per FUN_080b6c20). Without this the table's untouched init value (3) leaked into later f6/f8/medium-match copies whenever a match referenced a hash bucket only ever touched during a literal run — byte-correct through many blocks by chance, then producing sparse single-byte corruption the moment a genuinely-collided bucket got referenced. Root-caused by GDB-diffing the decoder's hash_table against ground truth at the exact block boundary where output first diverged. The "8/8 100%" figure is from the low-entropy coverage_matrix.sh fixture only and does NOT reflect real data — see the measured table in the README. Variant B hash init=3 fix (commit 049d041) closed the multi-file silent corruption.
lzpf prefilter+arith path (-cf) ✅ mono + stereo native FUN_080a5330 + LPC filter (FUN_08095d90) ported; mono AND stereo audio byte-exact. The stereo path was closed end-to-end (stereo_lms_cf.nz → stereo_lms.wav, both blocks byte-exact, -cf native_only_v2 9/10→10/10). The full FUN_080a5330 stereo flow is now reproduced in DecodePFBlock: residuals are PLANAR (ch1=[0,per_chan), ch2=[per_chan,2·per_chan)) decoded by two per-channel DecodeArithBuffer calls (each reads its own Huffman header — a single n_elems call mis-locates the side stream by ~1.4 KB); the predictor-init reads a leading bit G (iStack_50078, gates the inter-channel LMS) then per channel [active(1), order(3 if active)] (order = bits+8); per-channel LPC with two persistent predictor states (pred/pred2, threaded across blocks); the verified inter-channel LMS FUN_08096e20 (ApplyLmsInterChannel, regression test TestLmsInterChannel); and FUN_080a50c0 reconstruction (ReconstructStereoSamples: per-channel delta-integrate + L/R (channels==1) or mid/side (channels==2) interleave). -cF (lzpf B / nz_lzpf_large) stereo is also native: it uses the SAME FUN_080a4ea0 arith as -cf (not a vtable path), differing only in the LPC filter order — FUN_08095d90 obj+0x1c08 is 8-tap for variant B vs 4-tap for variant A (GDB-confirmed), so LpcPredictor is now N-tap (8-tap adapts every sample, 4-tap every other). -cF native_only_v2 8/10→9/10. A real file-format-sample sweep later found the single-fixture validation above had missed real bit-depth/channel variation (5/9 on a 9-file real stereo/multichannel WAV corpus spanning 2/6 channels, 8/16/24-bit): fixed 4 real bugs — tail-remainder bytes silently dropped instead of copied verbatim; the generic (non-int16-fast-path) reconstruction always did plain L/R interleave instead of mid/side for channels==2; the LMS per-object adaptation-shift bits were never read/applied; and LPC/LMS predictor state wasn't reset across a literal/LZ77 block sitting between prefilter blocks (GDB-confirmed on the real binary: state is exactly zero after every non-prefilter block, carried forward after every prefilter block, 16-for-16). Real-corpus audio sweep now 9/9 for both -cf and -cF.
lzhd decoder (-cd/-cD) ✅ LZ (both -cd raw-literal and -cD MTF+predictor literal models) + block-RLE + raw-store + pure-literal + exe + text-pipeline (param14/line-RLE/CRLF/word-dict) + multi-chunk text + large multi-stream files + parallel containers native; only rare CM/BWT sub-chunks bridge The real linux32 -cd is a coroutine token-LZ (NOT the reference DecLZ). Fully ported in nz_cd_tokens.{h,cpp} and validated byte-exact against the binary: token assembler (FUN_080aa070), reconstruction with trailing-literal flush (FUN_08099050), per-column RLE run-expander (FUN_080acb90, thr=1 for the LEN column / 0 for LIT·OFF), param14 text transform (FUN_080a0ff0), the bounded-varint header (FUN_080b1dc0), and the integrated chunk/stream/block decoders (NzCdDecodeLzChunk/NzCdDecodeStream/NzCdDecodeBlock). Columns/literals reuse DecodeArithBuffer. The reconstruction runs over a single per-archive ring (FUN_08099050, obj+0x978) whose size the encoder sets to round(total_output / 0x10000) · 0x10000 (min 64 KB) — GDB-confirmed across text50/source.cpp/big_code/repeat_3M = 1/3/19/46 × 64 KB. The size need not be a power of two, so the ring uses modular helpers (not a bitmask). The ring is sized to hold the whole compact recon, so its cursor (obj+0x980) advances monotonically and never wraps for real archives (the reset/wrap path is a safety fallback). Large files split their output into 1 MB streams; the ring is allocated once and persists across streams (NzCdDecodeStream threads ring_pos and the file-absolute output offset), so cross-stream matches resolve correctly. Wired into the extract dispatcher and BYTE-EXACT end-to-end under NZ_NO_BRIDGE=1 for: token-LZ (recon == file), block-RLE (flag &2, post-recon run-length re-expansion of collapsed zero-runs), raw-store (per-column b0&1==0 and flag-&1-clear literals = verbatim bytes), pure-literal (no LZ tokens — the whole window is one literal stream; generator picks this when size_field==0 or v2==0), exe (flag &4: a BCJ-style x86 E8/E9 call/jmp address un-transform, NzCdExeUnfilter), and the text pipeline (flag &8, FUN_080a3c90): param14 (NzCdParam14), line-RLE (FUN_080a2f20), CRLF EOL (FUN_080a19b0), and the word-dictionary transform (FUN_080a0a00 = the reference TransformText_1_Dictionary + dict/char-trait tables, in nz_cd_texttransform_dict.cpp). Verified on text (map.txt), binary (image.cat), an ELF (elf.bin, flags=3 block-RLE), an EXE (play.exe, flags=5), atoll (multi-chunk &8), word-dictionary text, multi-chunk text (text_50k/repeat_100k/CRLF), a 7-chunk source file (single-stream 192 KB ring), large multi-stream files (2–3.5 MB code/text/base64, incl. heavily-repetitive input), and a varied real corpus spanning images/audio/video/executables. As defense in depth TryDecodeLegacyLzhd self-verifies the decoded output against the archive's stored per-file checksum and returns false on mismatch, so any unforeseen edge falls through to a byte-exact bridge decode (no silent corruption, and NZ_NO_BRIDGE native-only is a provable correctness signal). The parallel multi-stream container (header flag 0x0f, >~8 MB, mirroring the -cf/-cF parallel format) also decodes byte-exact: N independent nz_cd streams, each with its own FRESH ring (unlike the single-container case, where one ring is shared across the whole archive — confirmed empirically: each parallel-encoder thread owns its own nz_cd instance), sized per-slice via the same round(slice_size/0x10000)·0x10000 formula; each stream's type-0 chunks are already-delimited raw DecLZ blocks (no concatenation needed, unlike lzpf), decoded one at a time into the slice and verified against its own per-slice checksum before being tiled into the assembled output. Verified byte-exact on parallel containers up to 20 MB (text, random, and mixed text+random content). -cD (nz_lzhds = "lzhd strong") shares this ENTIRE front end (same token assembler, same column decoding, same LZ match/rep-cache model — even the same FUN_08099050-equivalent recon loop, dispatched via a vtable slot) and differs only in the literal coder: a per-context MTF rank table (256 contexts × 64 bytes: 32-slot rank table + 32-byte presence bitmap, seeded to the identity permutation with bitmap bits 0–31 set, not all-zero) plus a 4-stage adaptive linear predictor (pred = (w3·h3+w2·h2+w0·h0+0x200+w1·h1)>>10, sign-updated toward the residual), fed by an MSB-first bit-reader + two-level Exp-Golomb integer decoder over a dedicated -cD-only bitstream (one raw length byte + that many raw bytes, sitting between the token assembler's extra-bits blob and the literal column in the chunk layout — GDB-confirmed, not documented anywhere in the archive format itself). Ported in nz_lzhds.{h,cpp} and wired into both the single-container and parallel-container -cD paths in TryDecodeLegacyLzhd. Remaining bridge cases: rare CM/BWT sub-chunks (shared with -cd, e.g. stereo audio). The old DecLZ (nz_lzhd.cpp, nzdec_v0) is the wrong format for linux32 -cd/-cD.
optimum decoder (-co/-cO) ✅ both native single-container AND parallel-container LZ / ⚠️ BWT still bridges Three RE sessions of GDB tracing against the real linux32/nz binary found the real -co/-cO decode core is not DecLZ (the community-reference codec that name was borrowed from is never called) — it is a distinct LZMA-shaped LZ77 engine with 4 repeat-offset slots (rep0–3), a unary/Elias-gamma-style length/distance code, and an adaptive literal mixer sharing its low-level binary range coder with nz_cm.cpp. -co (FUN_0809e600, the "compact" nz_optimum1 variant, 4-context literal mixer) ported in nz_optimum_lz.{h,cpp}; a real port bug (the literal mixer's context-C seed sign-extended a byte the real binary zero-extends before shifting) was found and fixed via GDB divergence-tracing against a real failing archive. Also ports NzTextTransformRle (tt_flags & 0x20) and NzTextTransformInsertLf (tt_flags & 0x02). Native for -co single-container AND parallel-container (flag 0x0f, >~8 MB — reuses -cf/-cd's already-solved chunk-record format verbatim) archives with decr_param==1 (LZ/CM) blocks; still bridges on decr_param==0 (BWT). -cO (FUN_080a5d90, the "large" nz_optimum2 variant) shares the same backbone but a materially richer 8-context literal mixer plus a rolling 3-byte-hash LZP-style secondary predictor with no -co analog — ported in nz_optimum2_lz.{h,cpp}, reusing nz_cm.cpp's own kLzModelLNext/CM_Input_Bit primitives and -co's own dumped tables directly (GDB-confirmed byte-identical, not re-embedded) rather than re-deriving them. Wired into TryDecodeLegacyOptimum for method_p0==6, checksum-gated identically to -co. Found and fixed 3 real port bugs via GDB ground-truth (a units-to-bytes conversion in the dispatch-bit's second APM stage, a missing counter-based addressing scheme in rep-slot selection, and a copy-paste wrong base address in the length decoder's extra-bits loop) plus a 4th, subtler divergence that only manifested once mixer/weight state moved past the cold-start case (needed a longer, real-world-shaped stress fixture to even surface). Native for -cO single-container AND parallel-container (flag 0x0f) archives with decr_param==1 blocks — the parallel-container branch is shared verbatim with -co (the decoder type is selected once via a small closure since it's fixed per-archive, not per-stream), no new RE needed. decr_param==0 (BWT) still bridges for both -co and -cO.
cm decoder (-cc) ✅ native byte-exact Native CM decoder (NZ_CM.cpp, 1100 LOC). The documented "byte-26" divergence is FIXED (2026-06-08): it was a one-line port error in CM_Input_Bit — factors0_err used truncating factors[0] / 16 instead of the reference arithmetic shift factors[0] >> 4 (differs for negative values, flipping the factor[7] zeroing condition). Found by a per-bit next_probability diff vs a compiled reference oracle. With the full post-filter pipeline ported — param2 RLE (NzBwtRleDecodeU32), param1 AddBytes delta filter (NzAddBytesFilter), tt08 dictionary, tt16 number-transform (NzTextTransformNumber) — plus multi-chunk decoding (CM state persists across consecutive type-0 chunks), stored blocks (param6==0 = raw payload, for incompressible data), and NzTextTransformInsertLf (tt_flags & 0x02, shared with -co), -cc decodes byte-exact natively (no bridge) on source, numbers, dates, IPs, prose, markdown, random/large multi-chunk: 13/13 on a comprehensive sweep, 9/10 on native_only_v2 (the one miss is the deferred stereo audio-CM variant). NzAddBytesFilter's BitReader had a latent word-boundary truncation bug (rounded its side-stream length down to a multiple of 4 bytes instead of using the exact byte count, matching the reference exactly except for this one truncation) that a real-world corpus sweep found — it silently failed (clean decline, never wrong bytes) on any real file whose param1 side-stream length wasn't a multiple of 4, undetected by the synthetic test corpus; fixed, and the fix transparently benefits -co/-cO too since they call the same function. Cross-chunk continuation across a stored block was also missing: when a param6==0 stored (raw) chunk sits between two decr_param==0 CM chunks, the CM model must observe those stored bytes (model-update only, via NzCmFeedByte) to stay in sync with the true output position for the next CM chunk — found on a real file (integrty.doc), fixed, -cc real-corpus 31→32/56.
Encode for all methods ⚠️ functional Native BWT/store/literal writers; the legacy compression bridge was disabled in commit 049d041 (IsInternalLegacyCompressionBridgeCompressor returns false unconditionally). Codecs the native encoder cannot handle now produce an explicit error. All 8 methods byte-exact round-trip via the native encoder for low-entropy inputs.

Fixture-based benchmark (coverage_matrix.sh)

The coverage_matrix.sh test uses a deterministic AES-CTR-of-zeros fixture (low entropy). On that fixture:

native_strict_percent = 100% (8/8) — no bridge, no compat, no original binary subprocess.

Low-entropy input rarely triggers the prefilter+arith block mode, so the fixture passes fully native even though that path is not yet ported. Real-world compressible data (text, source code, binaries) will hit that path and fall back to the extract bridge for -cf/-cF.

Important caveat: native_strict_percent measures "no [bridge] / [compat] log line in stdout". It does NOT measure "the C++ code path actually ran the native decode vs silently called the legacy binary via FindLegacyBackend". A fixture that triggers the prefilter+arith block mode shows the bridge being invoked at runtime but still reports native_strict_percent=100 because the bridge produces byte-exact output. To verify the code path is truly native, use tests/native_only_v2.sh (NZ_NO_BRIDGE=1) — see the README's measured table, which is the honest number.

Clone this wiki locally