Skip to content

1.3.0 — an xz stream the engine can see inside

Choose a tag to compare

@drsamonline drsamonline released this 07 Nov 14:52
· 29 commits to main since this release

An .xz stream the engine can see inside. Format stays v8 — but a stream from this build can carry a layer bit 1.2.0 does not know, and 1.2.0 refuses it. That is why this is a minor version and not a patch.

Added

  • .xz transform. A .tar.xz used to be opaque: the archive came out slightly larger than the stream. It is now inflated, and the stream is rebuilt from a recipe read out of its own block header — dictionary size, integrity type, a preset byte, one extreme bit — with nothing of the original stored.

    On a 522,240-byte tar: 156,904 → 130,449 bytes, against 130,435 for the tar itself. Byte-exact, costs one inflate plus one or two trial encodes, and the recipe was proved against streams written by a different liblzma version before any of it was built.

  • unknown-transform. A stream using a layer bit this build does not know now says so. Before it said bad-flags — "invalid header flags" — which reads as corruption in a file that is perfectly intact. Measured with a 1.2.0 binary against an .xz stream before changing anything.

  • Nested transforms inside an .xz. The .xz stage was not connected to the seven nested gates, so a .tar.xz holding a CSV or a ZIP was inflated and then left alone. The test for this fails without the fix — checked by narrowing the gates back.

Measured, and now checkable

  • Silesia, 211,938,580 bytes, both tools built from sources in the tree: ziptastic -9 gives 43,271,420 (20.42%) against an xz -9e equivalent at 48,456,004 (22.86%) — 10.7% smaller, 651 s at 0.33 MB/s. scripts/silesia.sh regenerates it, checks every file size before measuring, refuses to start without 500 MB free, and decodes each xz stream back before reporting its size. That last guard exists because a truncated write on a full /tmp once read as a 47% regression.
  • Per-file: reymont 22.1%, mr 18.2%, dickens 16.8%, osdb 16.7%, x-ray 15.4%, webster 14.5%, xml 13.8%, ooffice 12.2%, mozilla 8.7%, nci 1.0% — and two rows that are level rather than ahead, samba 0.1% behind and sao 0.2% ahead.

Still not built

.zst needs a decoder the tree does not have and a (level, framing) shortlist of four, because a stream written to a pipe omits its content size. .br is cheap in dependencies but quality 11 costs 2.5 s to confirm. bzip2 is measured at −9.7% with a one-byte recipe and parked for want of libbz2.

Checks

35 suites, 53 CLI assertions, scripts/regression-corpus.sh green — in CI, and on the artifact attached here: a 368 KB tar and its .tar.xz both round-tripped byte-exact, the latter with layers3 = 0x02.

ziptastic-1.3.0-linux-x86_64 is Linux x86-64, needs only glibc 2.17, and SHA256SUMS covers it.