Performance

Performance

Decode speed against the original, measured 2026-09-03 on the same machine (Xeon E5-2697A v4, 64 threads), t (test) mode, wall time, best of one run each. All changes below are byte-exact: the suites (release package 95/95, synthetic 96/96, multi-file 144+72, real corpus 488/488, damaged archives 42/48) did not move.

Mixed 137 MB tar (205 real files: text, documents, images, executables, audio, fonts), single container (-p1)

codec	original	nz_recon	ratio
-co nz_optimum1	19.1 s	14.2 s	0.74
-cO nz_optimum2	34.9 s	37.4 s	1.07
-cc nz_cm	146.1 s	170.2 s	1.16
-cd nz_lzhd	1.57 s	1.50 s	0.95
-cD nz_lzhds	1.71 s	2.26 s	1.32
-cf nz_lzpf	1.51 s	1.49 s	0.99

Large archives

archive	original	nz_recon
2.29 GB tar, -cf, 16 worker streams, t	4.75 s	4.0 s
2.29 GB tar, -cf, 16 worker streams, x (writes 2.29 GB)	4.6 s	5.5 s
400 MB synthetic, -cO -p1 (BWT + audio model blocks)	27.2 s	30.2 s
400 MB synthetic, -co, 8 streams, -t1	51.9 s	32.0 s

What the round changed (all found by profiling both binaries on the same archive)

• Fletcher16/32 reduced once per 64 KB block instead of per byte (was 14 % of a -cf decode).
• The trace/dump switches were read with getenv() inside per-symbol loops: 3–9 % of every decode. Cached per call site (NZ_ENV).
• The audio sign-LMS predictor is MMX in the original (paddsw/psubsw update, pmaddwd dot product); ours was scalar. Now SSE2, bit-identical by construction.
• The inverse BWT was one dependent chain of cache misses. The original (FUN_0809d370) walks many independent chains and stitches them from the primary index; ported, including its start-detection trick (chain id in the start index's low byte) and the cumulative-table symbol lookup, walked interleaved and on threads. The 256 BWT buckets, each with its own decoder, are decoded on threads too.
• -cd/-cD: the text pipeline allocated and zeroed two whole-output buffers per 32 KB chunk (89 % of the decode, 15.4 s vs 1.5 s). Persistent per-thread scratch, re-zeroed only where written.
• Parallel -cf/-cF streams decode straight into their slice of the output; the output buffer is no longer a zeroed std::vector (2.3 GB of memset on one thread before any worker could start, every page faulted twice). t on the 2.29 GB archive: 5.16 s → 4.0 s.
• The -cO mixer (dot product and weight update) is MMX in the original (pmaddwd, pmulhw/psraw/paddsw); ours was scalar with 64-bit multiplies. Now SSE2, bit-identical. The trace/dump switches on the per-bit path of both optimum cores compile in only with -DNZOPT_DEBUG (the write-watch hook with -DNZO2_WATCH). -cO 53.8 s → 37.4 s.

Where the remaining gap is

-cO: the LZP hash lookup alone is 12 % of the loop and the shared per-bit tail updates a dozen scratch cells in memory. -cc: the CM is inherently slow in both. -cD: the order-32 predictor. Output is still assembled whole in memory and written after the decode; the original writes each file while its streams decode, which is why it is faster on x of a huge file (and why the 32-bit builds cannot decode archives above 2 GB). A writer that consumes completed slices during the decode is the remaining structural step. Tooling and the profiling recipe (the original is stripped: samples are aggregated per Ghidra function) are kept with the project's private tooling under perf/.