Skip to content

1.4.0 — brotli and zstd, the other two containers

Choose a tag to compare

View all tags
@drsamonline drsamonline released this 08 Nov 09:59
· 22 commits to main since this release
v1.4.0
cbe064a

The other two containers. 1.3.0 taught the engine to see inside an .xz; this does the same for .br and .zst, which completes the set the Phase 4 survey found.

Added

  • A .br transform. Brotli has no magic number and does not record its quality, so the transform decodes, guesses the producing settings from a shortlist of six, and keeps the guess only if re-encoding reproduces the stream byte for byte. A quality-11 .tar.br of 163,436 bytes archives to 139,157, against 139,148 for the tar itself; at quality 1 the same transform is worth 37%.
  • A .zst transform. zstd has a magic number, but the level and the framing still have to be recovered. A level-19 .tar.zst of 170,767 archives to 139,155; at level 3 it is worth 30%, including the piped framing — which needed the encoder to feed its input in 128 KiB chunks, because a stream written to a pipe omits its content size and splits blocks where its writer chunked.

Both round-trip byte-exact on the attached artifact, with layers3 reading 0x04 and 0x08. Format stays v8, but the layer mask grew by two bits, so a 1.3.0 reader given such a stream refuses it — with unknown-transform, which says so, instead of "invalid header flags".

Measured, and closed

  • A real scanned PDF, 99 pages, 7,456,578 bytes: the engine gets 6,373,705 (85.5%), and precomp correctly declines. Four separate hypotheses were then measured and all four lost: CCITT G4 (5–8% worse than the Flate already in the file), this engine's own backends (40–48% worse on the sparse pages), a forced precomp+srep chain (8% worse), and the JBIG2 context-model bound (10–92% worse). The document is already close to optimally coded.
  • SWF dropped. Flash is dead and the measurement said there was nothing to win.

Checks

37 suites, 53 CLI assertions, scripts/regression-corpus.sh green — in CI and on the artifact: a 552,960-byte tar, its .tar.br, .tar.zst and .tar.xz all archived to within 13 bytes of each other and every one round-tripped byte-exact.

ziptastic-1.4.0-linux-x86_64 is Linux x86-64, needs only glibc 2.17, and SHA256SUMS covers it.

Assets 4
Loading