Skip to content

LowPack 0.2.0 — hostile-input hardening

Pre-release
Pre-release

Choose a tag to compare

@devkyato devkyato released this 29 Jul 11:08
b13d59c

I thought the right way to make this release was to review LowPack as if every input path and every archive field were trying to mislead it. That changed quite a bit under the surface, but the result is a much more honest and defensive alpha.

Oh! The important compatibility point first: v0.2.0 writes binary format 1.1 and manifest schema 2.0. The 0.1 format was explicitly experimental, and this release chooses stricter validation over pretending that unsafe relationships are compatible.

What changed

  • Direct inputs, traversed files, directories, and filesystem identities now share one collision registry.
  • Followed symlink cycles, duplicate identities, and output-inside-input recursion are rejected.
  • Source files are archived from one descriptor snapshot, hashing during the same chunk pass and rejecting detected mutation.
  • Dictionary sampling uses bounded reads; trained dictionaries are stored once and referenced by SHA-256 ID.
  • Codec explanations now separate the preferred per-file policy from the actual representation reused by deduplication.
  • Ordinary full verification and extraction stream one chunk at a time instead of caching complete archives or files.
  • Manifest schema 2.0 validates exact types, hashes, aggregate references, chunk boundaries and overlap, sizes, transforms, decisions, dictionaries, permissions, and total object count.
  • Canonical telemetry validates rows, columns, cells, bitmaps, indices, deltas, floats, and run lengths before reconstruction.
  • Archived permissions are opt-in and limited to ordinary rwx bits.
  • Selection goals use deterministic policy names instead of claiming measured speed.
  • Benchmarks now separate raw payload, deterministic tar-container, and full LowPack archive behavior.
  • CI includes hostile-input regressions, bounded-memory checks, 5,000 Atheris smoke runs, reproducible packaging, Windows coverage, and Node 24 action runtimes.

Verification

  • 62 tests collected; 59 passed and 3 expected platform skips on the release host
  • Ruff and strict Mypy passed
  • all ten pull-request and post-merge GitHub jobs passed
  • wheel and source archive reproduced byte-for-byte across clean isolated builds
  • the wheel passed a clean-install pack, quick/full verify, explain, unpack, and canonical telemetry smoke test

Current boundary

Canonical telemetry is still memory-resident, so I capped both its encoded and reconstructed forms at 64 MiB and added structural amplification checks. Ordinary files are streamed. For hostile extraction, use a new privately owned output directory that another untrusted process cannot modify concurrently.

The release remains marked as a prerelease because LowPack is still alpha.