1.4.1 — the audit's four fixes

Batch A of the plan: the audit's four fixes, all verified on the artifact rather than the build tree.

Fixed

• The compressor had no inflate cap of its own. The three container transforms capped inflation at a flat 4 GiB and the compressor allocated up to it, while the decompressor has always used a relative cap. A 28,971-byte .zst inflating to 900 MB took the old build down; it now declines and stores the stream opaquely, peaking at 524 MB instead of asking for 4 GiB.

  The cap is max(input × 4096, 256 MiB) in all three — the decoder's policy with a floor, so a 6 KB stream inflating to 200 MB (a tar of zeros) still transforms, and anything past it is declined rather than allocated.

• Base64 offered itself on a binary scanned PDF. The detector accepted any single run of 32 alphabet bytes, and a PDF dictionary supplies one; on a 7.4 MB scan that produced layers 0x10 and a wasted trial. It now requires the alphabet to cover at least 1/64 of the file — a real base64 file still fires, a 32-byte accident in 7 MB does not.

• A short encode now reports itself. brotli_wrap and zstd_wrap relied on the re-encode verifier to notice a stream that never reached FINISH. Both now say so where the cause is known, rather than leaving a truncated buffer to be diagnosed three functions away.

• expansion_cap is saturating rather than wrapping. It multiplies an attacker-supplied true_size; the old wrap rejected by accident, and now it cannot wrap at all.

Added

• A marker convention. TODO(name-date): what and why, documented in BUILD.md, with the six catch (...) blocks that swallow everything marked. There were zero markers in 12,460 lines before this, which is why the audit found three things a grep should have.

Checks

37 suites, 53 CLI assertions, CI green. Checked on the artifact itself: the 900 MB bomb declines at 524 MB peak, a real base64 file still sets layers 0x10, and a FLAC still gets no BCJ.

Format unchanged at v8; nothing here alters a stream's meaning, so 1.4.0 archives read identically.

1.4.0 — brotli and zstd, the other two containers

The other two containers. 1.3.0 taught the engine to see inside an .xz; this does the same for .br and .zst, which completes the set the Phase 4 survey found.

Added

• A .br transform. Brotli has no magic number and does not record its quality, so the transform decodes, guesses the producing settings from a shortlist of six, and keeps the guess only if re-encoding reproduces the stream byte for byte. A quality-11 .tar.br of 163,436 bytes archives to 139,157, against 139,148 for the tar itself; at quality 1 the same transform is worth 37%.
• A .zst transform. zstd has a magic number, but the level and the framing still have to be recovered. A level-19 .tar.zst of 170,767 archives to 139,155; at level 3 it is worth 30%, including the piped framing — which needed the encoder to feed its input in 128 KiB chunks, because a stream written to a pipe omits its content size and splits blocks where its writer chunked.

Both round-trip byte-exact on the attached artifact, with layers3 reading 0x04 and 0x08. Format stays v8, but the layer mask grew by two bits, so a 1.3.0 reader given such a stream refuses it — with unknown-transform, which says so, instead of "invalid header flags".

Measured, and closed

• A real scanned PDF, 99 pages, 7,456,578 bytes: the engine gets 6,373,705 (85.5%), and precomp correctly declines. Four separate hypotheses were then measured and all four lost: CCITT G4 (5–8% worse than the Flate already in the file), this engine's own backends (40–48% worse on the sparse pages), a forced precomp+srep chain (8% worse), and the JBIG2 context-model bound (10–92% worse). The document is already close to optimally coded.
• SWF dropped. Flash is dead and the measurement said there was nothing to win.

Checks

37 suites, 53 CLI assertions, scripts/regression-corpus.sh green — in CI and on the artifact: a 552,960-byte tar, its .tar.br, .tar.zst and .tar.xz all archived to within 13 bytes of each other and every one round-tripped byte-exact.

ziptastic-1.4.0-linux-x86_64 is Linux x86-64, needs only glibc 2.17, and SHA256SUMS covers it.

1.3.0 — an xz stream the engine can see inside

An .xz stream the engine can see inside. Format stays v8 — but a stream from this build can carry a layer bit 1.2.0 does not know, and 1.2.0 refuses it. That is why this is a minor version and not a patch.

Added

• .xz transform. A .tar.xz used to be opaque: the archive came out slightly larger than the stream. It is now inflated, and the stream is rebuilt from a recipe read out of its own block header — dictionary size, integrity type, a preset byte, one extreme bit — with nothing of the original stored.

  On a 522,240-byte tar: 156,904 → 130,449 bytes, against 130,435 for the tar itself. Byte-exact, costs one inflate plus one or two trial encodes, and the recipe was proved against streams written by a different liblzma version before any of it was built.

• unknown-transform. A stream using a layer bit this build does not know now says so. Before it said bad-flags — "invalid header flags" — which reads as corruption in a file that is perfectly intact. Measured with a 1.2.0 binary against an .xz stream before changing anything.

• Nested transforms inside an .xz. The .xz stage was not connected to the seven nested gates, so a .tar.xz holding a CSV or a ZIP was inflated and then left alone. The test for this fails without the fix — checked by narrowing the gates back.

Measured, and now checkable

• Silesia, 211,938,580 bytes, both tools built from sources in the tree: ziptastic -9 gives 43,271,420 (20.42%) against an xz -9e equivalent at 48,456,004 (22.86%) — 10.7% smaller, 651 s at 0.33 MB/s. scripts/silesia.sh regenerates it, checks every file size before measuring, refuses to start without 500 MB free, and decodes each xz stream back before reporting its size. That last guard exists because a truncated write on a full /tmp once read as a 47% regression.
• Per-file: reymont 22.1%, mr 18.2%, dickens 16.8%, osdb 16.7%, x-ray 15.4%, webster 14.5%, xml 13.8%, ooffice 12.2%, mozilla 8.7%, nci 1.0% — and two rows that are level rather than ahead, samba 0.1% behind and sao 0.2% ahead.

Still not built

.zst needs a decoder the tree does not have and a (level, framing) shortlist of four, because a stream written to a pipe omits its content size. .br is cheap in dependencies but quality 11 costs 2.5 s to confirm. bzip2 is measured at −9.7% with a one-byte recipe and parked for want of libbz2.

Checks

35 suites, 53 CLI assertions, scripts/regression-corpus.sh green — in CI, and on the artifact attached here: a 368 KB tar and its .tar.xz both round-tripped byte-exact, the latter with layers3 = 0x02.

ziptastic-1.3.0-linux-x86_64 is Linux x86-64, needs only glibc 2.17, and SHA256SUMS covers it.

1.2.0 — the same archives, less time

The same archives, less time. Every commit since 1.1.0 has been measurement or speed, and no archive size changed. Format stays v8: 1.1.0 and 1.2.0 read each other's archives.

What got faster

change	effect	archive
Huffman candidate gated by a 64 KiB trial	18% of a text run no longer spent on a candidate that never wins there	unchanged on six corpora
LZMA2 fallback gated with a 20% margin	6.6 s → 4.1 s on a 1.6 MB mixed tree	unchanged on six corpora
base64 gate: decisive 64 KiB first pass	1.76 s → 1.08 s where the answer is obvious	unchanged
block digests checked in the decode workers	strictly less serial work	unchanged

Each of those is a trial that measures, not a rule that predicts — this project has three failed predictors behind it and does not want a fourth.

What was measured and not built

• Block size swept at last (64 KiB to 16 MiB, four corpora): 4 MiB stays, 4.3–9.3% smaller than 256 KiB on homogeneous content, within 0.6% of the best everywhere.
• An adaptive block size rejected, with its mechanism measured: removing BWT widens the 0.6% gap to 6.7%, so it is not the selector's freedom alone, and the trial would cost a second encode of every block.
• Phase 4 opened: a .tar.zst costs 22.7% more through the engine than the tar itself; seeing inside zstd/brotli/xz is worth −18.5/−14.7/−16.9%. Recipe recovery settled as a shortlist, not a search (a search costs 2.5–4× the whole compression). Not built: it needs a corpus of real streams from other tools first.

One thing worth knowing

The development box reports two cores and delivers one core of throughput — two threads take as long as two sequential runs. Every thread claim in FINDINGS.md was measured there, so --parallel and -T are untested, not disproven. §0 of FINDINGS.md says so before the numbers it applies to.

Checks

34/34 suites, 53 CLI assertions, scripts/regression-corpus.sh green — on this build, in CI, and on the artifact attached here: a 1.6 MB asset tree round-tripped byte-exact at 319,947 bytes in 6.5 s.

ziptastic-1.2.0-linux-x86_64 is Linux x86-64, needs only glibc 2.17, and SHA256SUMS covers it. Full record: docs/FINDINGS.md.

1.1.0 — format v8, addressable blocks

Format v8: every block's position is now in the trailer, so a block can be reached without walking the stream.

This release cannot read 1.0.0 archives, and 1.0.0 cannot read these. v7 streams are refused with bad-version, which is this project's usual clean break. There are no v7 archives in the wild to strand.

What is new

• A block offset table in the trailer — one absolute u64 per block, behind a ZTOX tail marker so a reader finds the table by arithmetic from the end. The header is byte-identical to v7; only the trailer grew.
• ziptastic info — the header, the block table, and every block's offset, size and mode. It verifies the table against the manifest before printing anything.
• ziptastic d --block N — one block, decoded on its own.
• ziptastic d --from-block N — a decode that resumes at a block boundary.
• decode_blocks / decode_blocks_parallel in the library, for a caller that holds an index rather than a stream.
• Streamed ZIPs are reached — a ZIP written to a pipe puts its sizes in a data descriptor, and the scan read only local headers, so every member was skipped. 320,424 bytes before, 183,286 after (−42.8%); the same ZIP inside a tar 321,197 → 183,454 (−44.0%). This was recorded as "a ZIP inside a plain tar is not reached" and the tar was never the obstacle.

Numbers, measured

• Reading the index touches 0.76% of a 225-block, 1.38 MB archive, in 0.04 ms.
• One block decodes in 4–11 ms against ~800 ms for the whole stream.
• The table costs 8 × blocks + 4 bytes: 0.13% of that archive, 1.7% of a 687-byte one — recorded as a real cost, not rounded away.
• Ratio and speed are unchanged from 1.0.0: 20–36% smaller than xz -9e on a realistic asset tree, at 0.3–0.6 MB/s.

What it does not do

• Archive mode is still sequential. Its blocks may reference an earlier one, so --block on an archive-mode stream is a refusal by design, not a guess.
• A transformed stream cannot be sliced — the inverse chain needs the whole stream.
• The offset table is not a speed feature. An in-memory index-driven decode measured 4–6% faster and none of it was the table. FINDINGS.md §3c has the reasoning.

Checks

34 test suites, all green; 53 CLI assertions; scripts/regression-corpus.sh asserts the property each historical bug broke, not merely that the round trip works. Full record in docs/FINDINGS.md; the v8 design in docs/FORMAT_V8.md.

Assets

ziptastic-1.1.0-linux-x86_64 — Linux x86-64, statically linked against the vendored coders, built from this tag with the recipe in docs/BUILD.md. SHA256SUMS covers it. Source is GitHub's own archive of the tag; vendor/hcs/old also carries optional mirrors as submodules, none of which the build needs.