v0.17.4-pre — a checks the whole archive before keeping it

v0.17.4-pre — a checks the whole archive before keeping it

New: the whole-archive self-check

After a or w32c has written the last byte, and before keeping the archive,
nz-re now decodes the whole file and compares every entry with what it read
from disk (an archive being replaced is written beside it and renamed over it
only after the check passes): the same names, the same sizes and the same content (a
CRC-64 taken while each file was read, not the archive's own checksums, which
come from the same read and are absent under -hn).

It is there because of two earlier defects. Before, only each -co/-cO
block was decoded back as it was written, and that never saw the container:
the multi-file table defect of v0.15.2-pre to v0.16.0-pre and the parallel
worker header of v0.15.0-pre to v0.17.0-pre both went out as archives nothing
could read. Either would now be caught before the archive was kept.

• On success nothing changes: the archive is the same byte for byte, the
  console prints nothing extra, and the footer's time is still the
  compression's.
• On failure it prints Self-check failed: the archive does not read back (...); no archive was created. and writes nothing. An archive it was
  replacing stays as it was. The exit status is 0, as with every other
  message (quirk 1), and 1 under NZ_STRICT_EXIT=1.
• When it cannot run (not enough memory to read the archive back), the
  archive is kept, and a Note: line after the footer says it was not
  checked.
• Cost: a decode. On 12 MB of real files at -t1: -cn +14 %, -cf
  +59 %, -cF +100 %, -cd +34 %, -cD +17 %, -co +17 %, -cO +21 %,
  -cc +92 % (the CM decodes as slowly as it encodes). NZ_NO_SELFCHECK=1
  turns it off.
• s writes nothing, so it has nothing to check.

The original does no such check.

Fixed

An archive of no files did not open. When every input is excluded (-x)
or cannot be opened, a writes a small archive with no file in it (27 to 29
bytes, 33 to 37 with -p2), byte for byte what the original writes. The original's t and x read it as
Decompressed 0 bytes and its l as Total of 0 files, 0 bytes.; this port
said Legacy filename table encoding is not recognized. Now it matches, with
-pN too. Found because the new self-check would have thrown such an archive
away.

-cD: a read past the end of a buffer. The -cD decoder's bit reader
loaded whole 4-byte words and could read up to 3 bytes past the end of its
buffer. It was found by running the new self-check under AddressSanitizer, on
a valid archive (the same text file twice). No output changes: a well-formed
stream never uses those bytes.

-cc compression memory. The compressor's CM model was never freed. It
is now freed as soon as the archive is written, before the read-back needs
its own.

Policy

Every defect of the original is reproduced, as the maintainer has decided.
There are three permanent exceptions, because each would harm the user:

• the extractor never writes outside the directory it extracts into (the
  original follows ../ in a stored name, quirk 29);
• an archive of only empty files extracts, where the original crashes
  (quirk 28);
• at the end of input the overwrite question counts as No, where the original
  asks forever (quirk 3).

The README's temporary notice about the v0.15–v0.17.0 archives is gone.

Verification

Four static binaries (Linux and Windows, 64- and 32-bit) built from a fresh
clone of the tagged commit. The release package's 245 checks pass on all four,
the Windows two on a real Windows 10 machine.

• The self-check test (tests/encode/selfcheck.sh) passes 78/78 on both
  Linux binaries, and also under ASan and UBSan with no finding.
  • It is silent and changes nothing across 8 methods × 6 shapes and w32c.
  • Every injected fault is first shown to break the archive, then caught.
  • A failed replace leaves the old archive byte-identical.
• On Windows 10, with both .exes:
  • a replace that passes renames the new archive into place, and one that
    fails keeps the old one untouched, with no temporary file left;
  • -p3, -cc, w32c and the archive of no files all check clean.
• The encode oracle writes 135/135 archives byte-identical and
  cross-decodes all of them.

The other results:

• native decode 96/96; multi-file 144 trees and 72 listings; checksum modes
  240/240; self-extractors 8/8;
• stored_lz_block 3/3, multifile_sizes 40/40, parallel_readback 16/16,
  large_window 2/2, sink_files 28/28, multiblock_attrs 105/105,
  filter_checksum 8/8, the memory-line grid 76/76;
• pty_prompt_add 11/12, the end-of-input case being the one deliberate
  difference;
• the 45 images and the 127 mixed files give the same counts as v0.17.3-pre.

The damaged-archive suites and the truncation sweep give the same reports as
v0.17.3-pre. An ASan + UBSan fuzz run over 2268 corrupt inputs found nothing.

v0.17.3-pre — the stored LZ block, read right and written

v0.17.3-pre — the stored LZ block, read right and written

Fixed

Some -co archives made by the original did not decode. When the
original's LZ engine cannot shrink a block, it writes the block raw (a stored
LZ block). Its decoder copies that block into the window the way it writes any
decoded block, in 32 KB pieces; this port pushed it in through the window feed
the other block kinds use. When the stored block reaches the end of the
window's ring, or is nearly as large as the window, the two can leave the window
at different positions (they agree only if the 32 KB pieces happen to end where
the feed does, as in every archive tested before), and the next LZ block then
fails. That takes a small -m budget, or an
input that has already outgrown the window. t and x stopped there with
Archive corrupted. Error decoding (code 100). Nothing was ever extracted wrong
without that message. Found on a 12 MB folder of 21 corpus files, archived by the
original with a -co -t1 -m4m, which has a 578 206-byte stored block; confirmed
on the original under GDB. The release package now carries such an archive
(co_stlz2), and v0.17.2-pre fails it.

-co no longer refuses inputs that need a stored block. Where the original
stores a block, this port's compressor used to stop with This compressor cannot yet write this input. It now writes the stored block, as the original does, and
the original reads the result back. Among the inputs that were refused: the
folder above and PowerPacker.pp from the 127-file mixed corpus, now
byte-identical to the original's. After a stored block the original resets the
compressor's model and nothing else, and so does this port now. The same code,
reading and writing, serves -cO, although no -cO stored block turned up in
any file tried.

The folder itself still comes out 21 bytes shorter than the original's (5 with
-cO, which has no stored block there). The difference starts in one later
block's param14 pass, the same block under both methods, and is
quirk 72:
the original starts that pass from whatever an earlier one left in its scratch
memory.

Found on the way: quirk 78

With -r, the original wrote an archive of that folder that its own t rejects
(Error decoding (code 106), in the first block). The archive's bytes change
with the length of the output path, and every version is unreadable. The same
files named one by one on the command line give one readable archive, whatever
the path. So the original's compressor reads memory it never wrote, and the
heap left by the -r directory walk is what makes that read change the output.
This port does not reproduce it: with -r or without, it writes the same
readable archive.

Verification

Four static binaries (Linux and Windows, 64- and 32-bit) built from a fresh clone
of the tagged commit. The release package's 245 checks pass on all four, the
Windows two on a real Windows 10 machine. The new tests/encode/stored_lz_block.sh
passes 3/3 on both Linux binaries: -co, -cO and -cc archives of two corpus
files are byte-identical to the original's, and each program reads the other's.
On Windows 10 both .exes write that -co archive byte-identical to what the
Windows original writes on the same machine, and the original reads it.

The encode oracle writes 135/135 archives byte-identical and cross-decodes all of
them. Byte-identical to the original under -t1:

• the 45 images: -co 44, -cO 43, -cc, -cd and -cf 45;
• the 127 mixed files: -co 126 (one more than v0.17.2-pre), -cO 126, -cc 126;
• an 8 MB slice of a real .cab, with -co (two stored blocks) and -cO;
• the two inputs at the 8 MiB split threshold, and a 30 MB input, with -co.

The other results:

• native decode 96/96; multi-file 144 trees and 72 listings; checksum modes
  240/240; self-extractors 8/8;
• sink_files 28/28, multiblock_attrs 105/105, multifile_sizes 40/40,
  filter_checksum 8/8;
• parallel_readback 16/16, large_window 2/2, the memory-line grid 76/76;
• pty_prompt_add 11/12, the end-of-input case being the one deliberate
  difference.

The damaged-archive suites and the truncation sweep give the same reports as
v0.17.2-pre; in the truncation sweep, two -cd cuts get a different report
from the original itself from run to run. An ASan + UBSan fuzz run over 2268
corrupt inputs found nothing.

v0.17.2-pre — big windows decode, and a asks before overwriting

v0.17.2-pre — big windows decode, and a asks before overwriting

Fixed

-co/-cO archives with a window of 2 GiB or more crashed the decoder. The
ring was indexed with a signed 32-bit offset, so every position past 2 GiB read far
outside the buffer. The original writes such archives when one compressor gets a
large budget (-m above about 4 GB with -t1, with -cO, or on a single media
file), and its Win64 build decodes them. Checked on a real one: 2.4 GB compressed by
the Win64 original with -co -t1 -m8g (window 2.25 GiB) now decodes byte for byte
on the 64-bit builds, where v0.17.1-pre crashed at the first byte. The 32-bit builds
cannot hold such a window and say Out of memory!, as the original's 32-bit build
does.

a over an existing archive now asks first, as the original does (quirk 3):
Overwrite <archive> (Yes/No/Always)? right after the Archive: line. y or a
go on, n prints Cannot open archive! and leaves the archive alone, and -y
skips the question. At the end of input the answer counts as No; the original asks
forever there. This changes scripts: an nz-re a over an existing archive
without -y and with no terminal no longer replaces it.

Out of memory! where the original prints it. Right after the Threads: line,
before any compressor line, with status 255 and nothing written (measured on the
original's 32-bit build). It used to come after the compressor line with status 0.

Console. The -co/-cO compressor memory line (nz_optimum1 [N MB]) follows the
original's own method now, term by term: it was a fixed 18/34 MB, right only for a
budget of 16 MB or less. It matches the Linux 32-bit original in all 76 size and
budget combinations measured. At an 8 GB budget the Win64 build prints 1 MB less,
because its objects are sized for 64 bits. a and s print
Warning: only N MB free! when the budget exceeds the free memory the banner
shows, as the original does; this port never did.

The window limits, measured for a forum question: the window byte can express
32 GiB, but the original computes the window in 32 bits, so 4 GiB wraps to zero.
The Win64 original decodes windows of 3.875 GiB (-cc: exactly up to 2 GiB), and
its compressor never writes more than 3.75 GiB. The Linux 64-bit original no longer
starts on current kernels, which lack the vsyscall page it needs.

Verification

Four static binaries (Linux and Windows, 64- and 32-bit) built from a fresh clone
of the tagged commit. The release package's 243 checks on all four, the Windows two
on a real Windows 10. The 2.4 GB big-window archive decodes byte for byte on the
64-bit Linux build; a 3.75 GiB-window -co and -cO archive decodes on the Win64
build and gives Out of memory! with status 255 on both 32-bit builds. The encode
oracle writes 135/135 archives byte-identical and cross-decodes all of them. The
other results:

• parallel_readback 16/16, large_window 2/2;
• pty_prompt_add 11/12, the end-of-input case being the one deliberate
  difference;
• the memory-line grid 76/76;
• native decode 96/96; multi-file 144 trees and 72 listings; checksum modes
  240/240; self-extractors 8/8;
• sink_files 28/28, multiblock_attrs 105/105, multifile_sizes 40/40,
  filter_checksum 8/8.

The damaged-archive suites and the truncation sweep give the same reports as
v0.17.1-pre. An ASan + UBSan fuzz run over 2242 corrupt inputs found nothing.

v0.17.1-pre — a of a big input wrote archives nothing could read

v0.17.1-pre — a of a big input wrote archives nothing could read

⚠️ Check big or -pN archives written with v0.15.0-pre to v0.17.0-pre

On a machine with more than one thread, a of an input of 8 MB or more with
-co (the default), -cO or -cc could write an archive that neither this port
nor the original can read, and so did any a -co/-cO/-cc given -p2 or
more, at any size. From 8 MB of input (or whenever -pN asks for it) the work
is split across several compressors, and the header of the last one was written
short. The 8 MB is the input: the archive itself can be much smaller. The original
prints Internal error: 99723465! and, on Windows, then hangs; this port says
Archive corrupted. Error decoding (code 100). A plain
nz-re a archive.nz folder is enough when the folder holds 8 MB or more.

• -cc: from v0.15.0-pre. -co and -cO: from v0.15.1-pre. All three up to
  v0.17.0-pre. (v0.14.x declined such inputs, and sometimes -co still
  declined -- This compressor cannot yet write this input -- which wrote
  nothing.)
• Not affected: -cn, -cf, -cF, -cd, -cD; and, without an explicit
  -pN above 1, -t1 and inputs under 8 MB. (Measured: v0.17.0-pre a -co -t1 -p3 of 360 KB is unreadable too.)

Test big archives made with those releases with nz-re t archive.nz (or
nz t) and re-create any that fail from their source files with this release;
this release cannot read them either. xman found the same thing independently,
on Windows, while testing v0.17.0-pre.

Fixed

The parallel container header. The last worker's header is written ahead of
every other stream, and the block that wrote it still built the two-byte codec
record of the store era; -co/-cO need three bytes, -cc four. One header
writer now serves every stream. a -t1 -p3 and -p16 are byte-identical to the
original's for -co, -cO and -cc.

Which codecs split. Outside the media rule, the original gives one
compressor per thread to -cf, -cF, -co and plain -cd/-cD, and keeps a
single one for -cn, -cO, -cc and the -cdp/-cdP/-cDp/-cDP variants
(which use their threads inside one compressor); with 90 % or more of the input
in media files it gives min(files, threads) to every method, the store
included. This port split every compressing codec and never the store. Measured
at 20 MB and 16 threads: the original writes one nz_optimum2 [160 MB] and one
nz_cm [441 MB] where this port wrote sixteen. The compressor count now matches
the original in every case measured.

-cc with several compressors in one process, reading and writing. The CM
model chose its first hash slot before clearing its table, so the second and
later compressors of a container could start from what the previous one left in
memory. a -cc -t1 -p16 of 20 MB wrote a stream nobody could read, and
nz-re t -t1 refused a valid -cc -p12 archive made by the original (code
101). Whether it happened depended on the memory the allocator handed back;
valgrind found the line. Both now match the original.

a over an existing archive no longer deletes it when it declines. The
output was truncated before anything was compressed, and a compressor that
declines removes what it wrote, so a -r old.nz folder lost old.nz whenever
-co could not write the folder. The new archive is now written next to the old
one and renamed over it only when complete. (The original asks before
overwriting; this port still does not ask, see quirk 3.)

Smaller, all measured against the original: -cd of 126 MB or more in one
stream (-t1) was not the original's bytes (from that size the window reaches 128 MB, which gives the
original's match finder a second table; now identical, memory line included, up
to 328 MB tried); -co/-cO of fewer than 8 bytes declined; w32c with no
file arguments now archives the current directory, as a does; the -cf/-cF
compressor read two bytes in front of its window after a wrap (valgrind), which
was the allocator's header and now is a zeroed pad, as in the decoder.

Not fixed yet

• -co declines some inputs (This compressor cannot yet write this input):
  the stored LZ block form is not written. Nothing is written in that case, and
  an existing archive of the same name is now left alone.
• The [N MB] memory line of -co/-cO is a fixed 18/34 MB; the original's
  depends on -m (37/53 MB at -m64m). Cosmetic.
• Compression is single-threaded whatever -t says.

Verification

Four static binaries (Linux and Windows, 64- and 32-bit) built from a fresh clone
of the tagged commit. The release package's 243 checks on all four, the Windows
two on a real Windows 10, where a of 20 MB with the default threads (8) now
gives -co eight compressors and -cc one, both read by the Win64 original.
Encode oracle 135/135 byte-identical and cross-decoded; the new
tests/encode/parallel_readback.sh 16/16 (v0.17.0-pre: 6/16) and
tests/encode/large_window.sh 2/2 (v0.17.0-pre: 1/2); synthetic decode 96/96,
144 multi-file trees and 72 listings, 240/240 checksum-mode checks, 8/8
self-extracting archives, sink_files 28/28, multiblock_attrs 105/105,
multifile_sizes 40/40, filter_checksum 8/8. The damaged-archive suites
(single, and parallel: five -p4 fixtures x seven damages, 34/35 writing the
original's files) and the truncation sweep give the same reports as
v0.17.0-pre (the original's own report still varies on one cut -cd archive,
quirk 39). A valgrind sweep of all eight codecs, compressing and decompressing
-p3 containers, is clean, and the ASan + UBSan fuzz run over 2242 corrupt
inputs found nothing.

v0.17.0-pre — a wrote multi-file archives nothing could read

v0.17.0-pre — a wrote multi-file archives nothing could read

⚠️ Check archives written with v0.15.2-pre, v0.15.3-pre or v0.16.0-pre

a -co, a -cO and a -cc of several files could write an archive that
neither this port nor the original can read ("Archive corrupted. Unexpected
end of file."). -co is the default compressor, so a plain nz-re a archive.nz folder is affected. It takes a file that spans more than one output block
followed by other files -- two files of 3 MB are enough; an 18-file directory
tree of mixed sizes reproduced it too. Single-file archives are not affected,
nor are -cn, -cf, -cF, -cd and -cD.

The defect came in with v0.15.2-pre (2026-09-15). If you made multi-file
archives with any of those three releases, test them:

nz-re t archive.nz

and re-create any that fail from their source files with this release. The data
blocks inside such an archive are intact; only its file table repeats an entry,
so a salvage is possible in principle, but this release does not attempt it.

Fixed

a -co/-cO/-cc, multi-file (the warning above). The writer holds the
metadata a read produces until the next block is written, and an empty pending
range was left at [0,0) instead of its sentinel after a block that ended in the
middle of a file; the next merge then re-announced every entry from the first.
None of the tests had written an archive in which a file spans two blocks and
another follows: the encode oracle's multi-file inputs are small and every
corpus sweep writes one file per archive. tests/encode/multifile_sizes.sh
now writes five such shapes with all eight codecs -- 40/40 byte-identical to
the original and read back by it, at -t1 and at the default thread count;
4/40 were unreadable on v0.16.0-pre.

Extracting more than about a thousand files. Every output file stayed open
until the end, so a large extraction wrote the first files and printed Cannot write for the rest. On Linux the usual limit is 1024 descriptors (253 of 2000
files under a 256 limit); on Windows the C runtime stops at about 2048 (2045 of a
3000-file archive, measured on Windows 10). The original writes them all. It hit
every -co/-cO/-cc single-container archive and every parallel container.
Files are now closed as soon as they are complete; all 3000 come out on Windows
and all 2000 under the 256 limit on Linux.

-forceout and -sp could truncate a real file to 0 bytes. An empty entry
was created last, after the files that follow it, so its truncation landed on a
same-named file already written: x -forceout of a -co archive left a 0-byte
file where the original leaves the last entry's 2 000 000 bytes. Files are now
created and written in entry order.

Multi-file -t1 archives of several blocks were refused or mislabelled. A
block whose files are all mode 0600 has no permission record; reading the
records as one run per archive then refused them all, so -cf called an intact
archive corrupt and -cd/-cc gave every file mode 0664 and a date in 1997.
tests/parity/multiblock_attrs.sh: 105/105 archives with files and listing
identical to the original, against 69/105 before. The original's l lists such
an archive's modes shifted by one block while its x writes them correctly;
that is reproduced, as quirk 76.

A file filter decides what is written, not what is checked. With the middle
entry's stored checksum damaged, x archive.nz a.txt of the original writes
a.txt alone and still prints the mismatch line for the other entry; this port
did not.

Smaller parity fixes, all measured on the original: an empty file's damaged
checksum is reported in x and not in t; a full disk prints Out of disk space! once and extraction goes on with the other files (it used to be
silent); [...] in a filename argument is a character class; -co/-cO
param14 no longer starts from zeroed tables, which had filled every empty
bucket with phantom candidates.

Closer to the original's bytes

The -co/-cO match finder's tree was half the size it should be, because the
encoder never told it the block size. Sized correctly:

	v0.16.0-pre	this release
45 BMP/TGA/TIFF/PNM, -co / -cO	40 / 40	44 / 43
127 mixed files, -co / -cO / -cc	123 / 124 / 126	125 / 126 / 126

The files still different all round-trip; the traced ones come from
quirk 72, the original starting each pass on a
buffer's leftovers, which is left unreproduced on purpose.

Faster, and less memory

Measured against the original on one thread; every archive byte-identical
before and after each change.

• Compression, -co/-cO. The BWT rotation sort is a linear-time SA-IS
  (prefix doubling kept for periodic blocks, where the two orders differ), the
  bucket coder walks its state machine once instead of twice, and the entropy
  segmenter no longer calls getenv() once per byte. On 3 MB of text -co went
  from 1.93x the original's time to about 1.4x, -cO from 1.73x to about 1.2x.
• Decompression memory. On 128 MB of real files: -cd 765 → 359 MB, -cD
  759 → 352 MB, -cf 433 → 179 MB, -cF 492 → 239 MB (the original: 109, 109,
  133 and 195 MB). -cf/-cF single containers now stream to disk one data
  record at a time, the unit the original uses.

Single-threaded, this port is still slower than the original. On 128 MB of
mixed real files (-t1, best of two runs, time as a multiple of the original's):

	-cn	-cf	-cF	-cd	-cD	-co	-cO	-cc
compress	1.70x	1.65x	1.64x	1.47x	1.80x	1.85x	1.54x	1.29x
decompress	1.02x	1.16x	0.99x	1.59x	1.41x	1.31x	1.19x	1.08x

Compression is slower on every method; decompression is level with the original
on -cn and -cF and slower on the rest. An earlier claim that it was faster on
-cf, -cd and -cc came from a fixture too small to measure anything but
start-up, and it was wrong.

Verification

Four static binaries (Linux and Windows, 64- and 32-bit) built from a fresh clone
of the tagged commit. The release package's 243 checks on all four, the Windows
two on a real Windows 10 machine. Encode oracle 135/135 byte-identical and
cross-decoded, synthetic decode 96/96, 144 multi-file trees and 72 listings,
240/240 checksum-mode checks, 8/8 self-extracting archives, the damaged-archive
suites (single and parallel) identical to the previous release, and the ASan +
UBSan fuzz run over 2242 corrupt inputs with no finding. New:
tests/encode/multifile_sizes.sh, tests/parity/sink_files.sh,
tests/parity/multiblock_attrs.sh, tests/parity/filter_checksum.sh -- each
fails on the previous release.

v0.16.0-pre — the last two block kinds the compressor could not write

v0.16.0-pre — the last two block kinds the compressor could not write

a -co, a -cO and a -cc now write audio and image blocks, the two
block kinds the original's optimum writers produce that this port could only
read. With them the three optimum codecs stop falling back to LZ on the files
those blocks exist for, and the archives come out byte-identical to the
original's on far more of them.

Nothing about decoding changes, and a build of v0.15.3-pre reads everything this
one writes.

What that is worth, measured

Two corpora, every file compressed by the original and by this port with the same
switches and compared byte for byte:

	v0.15.3-pre	this release
45 BMP/TGA/TIFF/PNM, -co / -cO	35 / 34	40 / 40
the same 45, -cc	40	45 — all of them
the same 45, -cn -cf -cF -cd -cD	45 each	45 each
127 mixed files, -co / -cO / -cc	118 / 118 / 120	123 / 124 / 126

The oracle suite is 135/135 byte-identical both ways, 144 multi-file trees and 72
listings are byte-exact, and every archive this port writes was extracted by
the original's own binary and compared against the source.

Two other fixes in the same range

A block the bucket coder should have compressed went out stored. Its capacity
argument is block_size * 5 + 0x100487, and this port passed the constant
0x600487 — that same formula frozen at a 1 MB block. Any larger block hit the
coder's own guard and was written raw:

	v0.15.3-pre	this release
a 1.9 MB .pbm, -cO	1 892 160 bytes	105 900, byte-identical to the original
a 1.97 MB .tif, -cO	1 801 177 bytes	1 387 673, byte-identical

An absolute path is stored without its leading /. a x.nz /etc/hostname
stores the entry as etc/hostname in the original; this port kept the slash, so
the archive differed from the first byte of the name and an extraction would have
written outside the current directory. No test covered an absolute argument,
which is why it lasted this long.

Found while wiring the new block kinds, never published

Both of these were defects in the audio writer, which is new in this release, so
no archive any release wrote is affected. They are worth naming because of how
they showed up: the archives decoded to the wrong bytes in the original's own
binary — silent corruption, not a refusal.

• FUN_08081c40 refuses a chunk whose payload does not come out strictly below
  the aligned sample bytes, and one refused chunk refuses the whole block. Without
  it this port wrote audio blocks the original codes as LZ.
• the audio model is reset on both sides of the decision — before coding when no
  span is in progress (that same flag is the mode2_type byte the block
  carries) and after any block not written as audio. So it carries across
  consecutive audio blocks only, which is exactly what the decoder assumes.

And one rule that is easy to get backwards: the signed_ flag the format probe
carries says whether a sample is read sign-extended. The decoder never needs
it — it writes back the low byte either way — but it decides every delta on the
encoder's side, and the two readings differ by a whole 2^width on any sample past
the halfway point. A .SOU and a .ss, both 8-bit mono, pick opposite ways.

What is still pending

• The optimum parser still differs on a few files. 5 of the 45 images and 2
  of the 127 mixed files come out a little larger or smaller than the original's
  under -co/-cO. It is a parse decision, never wrong data: those archives
  round-trip. Traced this week down to the match finder's tree — at one position
  the original has an 11-byte candidate that our chain walk, which ends after
  eight steps, never reaches.
• a writes this project's own container. The original rejects it as
  "incompatible version (0.90)".
• A multi-file archive above roughly 8 MB of total input becomes a parallel
  container and is declined on decode; a single-file archive of any size is fine.
• Given a filename containing [...], the original treats the brackets as a glob
  character class and finds nothing where this port matches the name literally.

Verification

Built from a fresh clone of the tagged commit, four static binaries (Linux and
Windows, x64 and x86). Each one extracts the release package's 242 expected
hashes, on the default path with no environment variables set; the Windows
binaries are checked on a real Windows 10 VM, not only cross-compiled. The
fuzz suite runs before tagging.

v0.15.3-pre — the banner counted threads where the original counts cores

v0.15.3-pre — the banner counted threads where the original counts cores

A one-bug release. Nothing about archives changes: the binaries of v0.15.2-pre
and of this one write the same bytes, and a build of either reads the other's
output. What changes is the second line every command prints.

#<n> is physical cores

Reported, in effect, by a user's own paste of three runs on one machine — the
original beside two of ours, on an 8-core/16-thread Xeon E5-2665:

original       ...|28688 MHz|#8+HT|4096/4096 MB
ours, v0.14.2  ...|1018 MHz|#16+HT|4096/4096 MB

#<n> is the count of physical cores. The Windows path printed the logical
one, so every hyper-threaded machine saw twice the right number.

Two details are what made it survive a dozen releases:

• the total has to come from GetSystemInfo and the hyper-threading ratio from
  CPUID, because the OS can be wrong about the topology where it is right
  about the total — under wine GetLogicalProcessorInformation describes this
  32-core/64-thread host as 32 cores of one thread each, while GetSystemInfo
  still says 64;
• unlike Threads:, the figure is not capped at 32.

Checked against the original's own Windows build:

	banner
the original (Win32, under wine, 32c/64t host)	…|2591 MHz|#32+HT|4096/4096 MB
v0.15.2-pre, nz-re-windows-x86.exe	#32 — and no +HT
this release	#32+HT

On the reporter's machine it now prints #8+HT. Threads: is unchanged (16 on
this host, matching the original) and so is the Linux banner, which was right
there for the wrong reason — it capped the logical count at 32 on a host whose
physical count is also 32 — and now derives the same figure the same way
Windows does.

The one thing still wrong on that line is the MHz field, which is a measured
figure rather than a clock (the original reports 28688 on a 2.4 GHz part). Ours
agrees with it to a few MHz here and not at all on the reporter's machine, and
until that is understood it stays as it is.

Verified

Decode, 243 archives x 4 binaries	243/243 each; the two .exe on a real Windows 10
The patched Windows builds vs the published ones, same input	archives byte-identical
Banner and Threads: against the original	identical on Linux natively and on both Windows builds under wine
native_only_v2	96/96
multifile_v2	144 trees + 72 listings
Corrupt and truncated archives	fuzz, 0 sanitizer findings / 0 timeouts / 0 crashes

What is still pending

Everything v0.15.2-pre listed. One item that looked new while checking this fix
turned out not to be ours: with more than one thread the original's own output
is not reproducible. The reader and the compressor run concurrently, so how
many files the reader has in hand when the compressor takes a piece depends on
timing, and the per-file records are grouped differently from run to run. The
same two-file input gave the original 425 590 bytes on some runs and 425 599 on
others; under -t1 it is always 425 599, and so is ours, byte for byte, in all
eight codecs. That is quirk 58 in docs/ORIGINAL_QUIRKS.md, and it is worth
repeating here because it makes any byte comparison against the original flap
unless both sides run single-threaded.

v0.15.2-pre — every input over a megabyte was cut in the wrong places

v0.15.2-pre — every input over a megabyte was cut in the wrong places

A fix release for the three compressors that write with a parser (-co, -cO,
-cc), plus a speed round on the same code. If you compress anything larger
than a megabyte, or several files at once, this is the one to take.

Where a block ends

The driver that decides how long a block is, FUN_0808d0b0, is not a function
of the bytes it is handed. It keeps a 1 MB staging buffer, reads from the
stream until that buffer holds 0xff000 bytes — or until the stream ends — and
only then asks the segmenter where to cut. What the cut leaves over is pushed
back into the buffer and starts the next block.

This port was cutting each 1 MB piece on its own, which put a block boundary at
every megabyte. A 1.1 MB file came out as

691274 / 64391 / 236926 / 43493 / 46194 / 5230     v0.15.1 (six blocks)
691274 / 313809 / 89687 / 5230                     the original (four)

and every input above a megabyte differed from its first boundary on. The whole
function is ported now, including the multi-round path (a block that swallows
the buffer goes back for more input and re-primes the segmenter with the halves
folded together) and the quirk that a tail too large to give back leaves the
block whole.

Two consequences of the same shape, both visible in the archive:

• the reader is asked for 0x100000 - held, not a flat megabyte, so a
  file's metadata records fall where the read that finished it fell. A 1.5 MB
  file's checksum was coming out one block early;
• those records wait for the next block. Two files that end before the same
  block are announced in ONE type-5 record of eight bytes, not two of four.

And -cc's block size is a flat 1 MB — 0x0804d173 hands the sizing helper
the constant — not the formula the two optimum engines use. The two agree on
every input up to a megabyte, which is why only a 1.1 MB file ever showed it:
one byte of the parameter record.

Speed

perf on a 4 MB -cO encode put 80 % of the run inside the parser, and almost
all of that in two loops that cleared the whole 133 KB dynamic-programming node
array — which the function does once per handful of input bytes. It now clears
only the part the previous call dirtied. Same bytes out, measured against the
original on the same 4 MB mixed input:

	v0.15.1	v0.15.2	the original
a -co	2.86 s	2.28 s	1.08 s
a -cO	11.77 s	3.83 s	2.14 s
a -cc	5.15 s	4.44 s	4.29 s

That is the half of xman's "sometimes 4–5 times slower than the original"
that belonged to us. What is left is the parse itself and the verifier decode.

Verified

Decode, 243 archives x 4 binaries	243/243 each; the Windows two on a real Windows 10
Encode, 294 corpus files x 3 codecs (-co/-cO/-cc)	292 / 292 / 294 byte-identical, 0 declined
Encode, the same corpus in groups of four files	45 / 44 / 45 of 45 (v0.15.1: 38 / 37 / 38)
Encode, six inputs of 1.5 to 4 MB x 3 codecs	10 of 18 byte-identical (v0.15.1: none)
native_only_v2	96/96
multifile_v2	144 trees + 72 listings
Corrupt and truncated archives	fuzz, 0 sanitizer findings / 0 timeouts / 0 crashes

What is still pending

The eight big inputs that still differ, 150_menu.tbk_, and one group of four
under -cO all share one cause, now identified: the parser's match-finder
tree and param14's hash table are the same 3 MB of memory in the original
(param14's head is 2^19 words and its chain 2^18 — 2 MB + 1 MB, exactly the
blocksize * 3 the tree is given). Each pass therefore starts on whatever the
other left behind, and this port's buffers are private and clean. It is quirk 72
in docs/ORIGINAL_QUIRKS.md, with the layout written down; reproducing it is a
change of where the buffers live, not of what the code does.

Unchanged from 0.15.0: the image and audio block kinds and the stored LZ form
are not written, so a file the original would code that way still comes out as a
valid archive it reads back byte-exact, just not the same bytes (a 108 KB .wav:
68 366 against its 67 614). A block that cannot be written at all is declined
with a message and no archive is left behind, and every block that is written is
proved readable by our own decoder before it is committed.

v0.15.1-pre — the three things v0.15.0 got wrong on Windows and with more than one file

v0.15.1-pre — the three things v0.15.0 got wrong on Windows and with more than one file

A fix release. Everything here was found by xman testing v0.15.0-pre, or
while chasing what he found. If you are on v0.15.0-pre and use Windows, or pass
a more than one file, this is the one to take.

Accented filenames, twice over

"Characters with diacritics in the unpacked file differ from the original."

Reproduced on a Windows 10 with a CP1252 machine code page, four files named in
Hungarian. It was two independent bugs.

Extraction wrote the name as UTF-8. The file creation called
_open(path.string()), and on MinGW path::string() is UTF-8 while _open is
the ANSI entry point — so ö reached the filesystem as the two characters its
UTF-8 form spells in CP1252:

Köntösömre sorsot vetnek robzombi nyers.mpg     <- the original
Köntösömre sorsot vetnek robzombi nyers.mpg  <- v0.15.0

Only the codecs that extract through that sink were affected — -co, -cO,
-cc — which is why -cn and -cd looked fine and the bug survived a release.

Compression stored a file the original refuses, under a mangled name.
WideCharToMultiByte without WC_NO_BEST_FIT_CHARS silently APPROXIMATES: on
CP1252 the Hungarian ű and ő come back as plain u and o, so the name
looked convertible and árvíztűrő.txt went into the archive as
arvizturo.txt. The Win32 A file APIs do not best-fit filenames, which is
exactly why the original cannot open that file and stores nothing at all.

a with more than one file

"Opt1 creates archives (and accepts input data) of up to 1 MB, typically
handling a single file. It processes two files but fails with three."

Both halves were one bug, and it was ours. The driver hands a compressing codec
one piece of at most 1 MB at a time, so a multi-file archive — or any single
file over 1 MB — gets several calls. Two pieces of state that belong to the
STREAM were being rebuilt on every call:

• the verifier, the second engine that proves each block readable before it
  is committed. From the second piece on it was cold, could not read what the
  warm encoder had just written, and the writer correctly refused to leave an
  archive behind. That was the "This compressor cannot yet write this input";
• the dece filter's run state, so a run of filtered blocks spanning two
  pieces reset in the middle.

A third one came out of the same test: the optimum and CM codecs do not stop
their 1 MB piece at a file boundary at all, and we were cutting it at 64 KB —
every multi-file archive diverged from the original's at the first filename
table.

Groups of four consecutive corpus files in one a, 45 groups, against the
original:

	v0.15.0	v0.15.1
-cn -cf -cF	45/45	45/45
-cd -cD	44/45	44/45
-co	declined	38/45
-cO	declined	37/45
-cc	0/45	38/45
declined	all three	none

What still differs there is not about multi-file: every file of those groups is
byte-exact on its own. They share one cause — the text pipeline's dictionary
step returns a slightly different size on large blocks (619 177 bytes in,
508 341 against the original's 508 309) — which is next, and which is also why
an input over 1 MB now produces an archive rather than a refusal, but not yet
the original's bytes.

The overwrite prompt

Visible in the screenshot xman posted of the two runs side by side: the original
asks

Overwrite ...ömre sorsot vetnek robzombi nyers.mpg (Yes/No/Always)?

and we asked about all 42 characters. The prompt uses the progress line's own
elision — ... plus the last 37 characters for a name over 40. Fixed in both
extraction paths.

Verified

Decode, 243 archives x 4 binaries	243/243 each; the Windows two on a real Windows 10
Encode, 294 corpus files x 3 codecs (-co/-cO/-cc), both Linux binaries	292 / 292 / 294 byte-identical, 0 declined
Encode on Windows against the ORIGINAL's own Windows build, same directory	21/21 on both binaries
Accented names, 3 codecs, archive AND extracted names	identical to the original, both binaries
CLI matrix, 35 cases	only argv[0] in the usage line and a progress interleaving differ, both known
native_only_v2	96/96

Nothing else changed: same codecs, same formats, same everything the 0.15.0
notes describe.

v0.15.0-pre — every compressor writes now

v0.15.0-pre — every compressor writes now

a -cO and a -cc join a -co. There is no compressor left that a refuses,
and for all three the archive is the original's own bytes on almost everything
tested.

a -cN -t1 -m4m, 294 corpus files, original against ours	byte-identical	declined
-co	292	0
-cO	292	0
-cc	294	0

The two -co/-cO differences are the same two files in both, both already
written down: 081_AWSOFTWA.PLA_ (the encoder's parameter buffer carries 256
bytes of a previous block's residue, and reproducing it needs the original's
ping-pong buffer pair) and 150_menu.tbk_ (an LZ parse divergence, ours
consistently 30-40 bytes smaller).

-cO

The block loop became one template over the bit source: a decoding IO reads
each bit from the range decoder, an encoding one codes the bit it was handed, and
every model read, update and symbol assembly between them is written once. Twelve
bit sites, each fed from the decision it belongs to. NZO2_RECODE=1 decodes a
real block recording its decisions and re-encodes them from the state the block
started in: 451 blocks of 294 archives, all byte-identical.

Then the parser (FUN_08083e90) — the structural twin of -co's, with a
0x1040-node horizon instead of 0x120, a match taken whole at 0x200 bytes instead
of 0x20, a match finder walked 0x100 chain steps instead of 0x10, and a literal
priced straight from the eight-input mixer with no cache at all.

It disagreed with the original after a hundred or so decisions, and everything
looked right: the finder's configuration read live out of the running original,
its candidate lists, every price formula. The reason turned out to be one line
in the coding loop, not in the parser — coding a match writes 0xffff over that
length's cached price, so the next parse prices it against the model the match
just advanced. Without it the cache only ever decays, and a length the original
charged 147 for we charged 315. With it, 110 of 111 blocks parse to the
original's own decision list with a byte-identical payload.

-cc

CM_Input_Bit never reads the arithmetic coder — it takes a bit and updates the
model — so the encoder is the decode loop with the range decoder replaced by a
range coder fed the bit the output byte carries. Nothing else changes, which is
why the two sides stay in step by construction. The flush is one byte: four
made every payload exactly three too long with the first 394 identical. 513
blocks of 294 archives re-code to the original's own payload byte for byte.

Its sizing is its own branch of the allocator: 26 MB off the budget, never below
5 MB, a quarter of the rest is the window's first claim, and when the input is
smaller than that quarter the window becomes the input and the two hash tables
take the rest — otherwise the tables are sized first and the window is what they
leave. Read live out of the original's own codec object on a 7 x 13 grid of input
sizes and budgets: 91 of 91.

Two things this found along the way

A Windows path bug, in every codec, since the encoder first shipped. The file
scan split arguments on / alone, so the separator a Windows user actually types
matched nothing:

> nz-re.exe a out.nz in\file.bin
No files found with in\file.bin
Nothing to do (no files found).

The original takes it and stores the entry as in/file.bin; so do we now.

A binary built from source ran four to five times slower than the released
one — reported by xman, who built the 32-bit target himself. A plain
cmake -B build emitted no optimisation flag at all; the released binaries come
from a script that passes -O2 -DNDEBUG explicitly, so the gap never showed
here. Measured on the same 29 MB archive: 27.08 s against 5.70 s, where the
original takes 5.67 s. CMAKE_BUILD_TYPE now defaults to Release. If you built
from source before, that alone was the difference.

Not written yet

The image and audio block kinds, and the stored LZ form. A file the original
would code that way still comes out as a valid archive — the ORIGINAL reads it
back byte-exact — just not the same bytes (a 108 KB .wav: 68 366 against its
67 614). A block that cannot be written at all is declined with a message and no
archive is left behind, and every block that is written is proved readable by our
own decoder before it is committed.

Verified

Decode, 243 archives x 4 binaries (Linux x64/x86, Windows x64/x86)	243/243 each, the Windows two run on a real Windows 10
Encode, 294 corpus files x 3 codecs x 2 Linux binaries	as the table above, identical on x64 and x86
Encode on Windows, 7 files x 3 codecs x 2 binaries, against the ORIGINAL's own Windows build in the same directory	21/21 each
Budgets 16 MB / 64 MB / 256 MB, 3 codecs x 5 files	45/45 byte-identical
native_only_v2 (byte-exact native decodes, zero bridge)	96/96
Console for a and s, four budgets, three codecs	identical to the original