v0.24.4

v0.24.4 - 2026-11-04

Added

• Mutation testing of the security-critical modules (secret masking, workflow commands, expressions, the native store, minisign): runner/tools/mutate.py changes a module's code one thing at a time (a comparison flipped, and for or, a return error.X guard taken away, a constant off by one, a statement deleted), runs that module's tests on each change, and reports every change they let through by file and line. The mutation workflow runs it for one module when asked for, over four of GitHub's runners, with the report as an artifact; it is too slow for a pull request. New tests kill the most important survivors of the first run: in masking (a lone secret, a one-byte one, a line of exactly four bytes, URL-encoded forms, bytes past ASCII, the stop-commands token), the native store (collect keeping what it should and evicting least recently used first, a changed record, half-built layers, record limits), workflow commands (an escape cut short at the end of the data, which a mutant read past), minisign (the key's algorithm tag, the trusted comment's prefix and limit), and if: conditions after a failure.
• Line coverage of the runner's tests with kcov (Ubuntu's package, pinned), nightly and on every pull request that changes runner/src, as analysis's "Coverage" job: the rootless tests, the native backend's root pass and the runner binary they start for their sandboxes, merged into an HTML report with Cobertura XML and JSON (the coverage artifact), and a job summary with the total, each directory and the least-covered security-relevant files. It reports without failing; COVERAGE_MIN sets a floor later. Nothing is sent to a coverage service.
• Store stress runs on a pull request that changes the native store, its layers, or the locks and mounts around them (the new store scope), not only nightly.

Fixed

• The native layer store refuses to publish a layer unless its key's lock is held, so a caller that forgets it (as Dockerfile builds once did) fails at once instead of racing another runner's collect. A new stress test runs six processes against one native store under a small cap for 30 seconds (rootless and as root, on x86_64 and arm64; ten minutes with eight processes in the nightly analysis workflow), each running jobs and Dockerfile builds on overlapping keys, and checks that every replay matches the live run that stored it, no job sees an evicted or half-published layer, every entry stays whole, no key runs in two jobs at once, no fd or lock leaks and no warning is logged, and a last collect leaves no temporary entry and the store within its cap. It found the Dockerfile build lock race independently of the store's TLA+ model.
• A native job's directory is no longer removed by another runner while its own runner is removing it: the job's lock file went early in that removal, and another runner sharing the store, starting a job, made a new lock in its place, found it free, took the directory for a dead runner's and removed it at the same time (logging AccessDenied when the two collided). A job's directory is now removed with its lock last, and a runner's sweep opens a lock without making one; a directory a dead runner really left is removed with a warning. Found by the store's stress test in its first ten-minute nightly run.
• The native layer store counts a layer's bytes once it is on disk: it measured the tree before its syncfs, while the kernel could still be writing back a file just written, and counted a block short now and then (seen on x86_64 and arm64, rootless and as root), so the store could hold a little more than its cap. The tree itself was always whole. And a Dockerfile build looks its layer up under the key's lock before publishing, as before the store's TLA+ model work: publishing first, a collect could evict the layer another builder stored between the refused publish and the lookup, and the build failed for nothing. The model now checks that no build is lost so (NoBuildLost), with the publish-first order as a mutant it must reject.
• A native job cancelled while a step waited for its key's lock (another job running the same step) or for a service to be ready now stops: both waits swallowed the cancel, which std.Io delivers once, and went on until the other job finished or the service's readiness limit passed. ZLint's suppressed-errors rule, which ci cannot run until ZLint applies line-level disables, now runs nightly (analysis's "ZLint (suppressed-errors)") against a baseline of the 119 sites read and meant, matched by file and a fingerprint of the line, and fails on a new one. CI's Workflows job also runs hadolint on the Dockerfile (whose findings are fixed: one RUN for the certs stage, pipefail for the build stage's; unversioned apk packages kept, with why) and shfmt on every shell script, both pinned and checked against their checksums, with Renovate following them.
• Native steps no longer wait for every other write to the disk they run on, which cost native jobs seconds on razorbill's SD card (toolcache 26 s and cache 20 s against Dagger's 6 s and 8 s), most of all while another job filled its tool cache. Each step's overlay synced its whole upper filesystem when it was unmounted, at every step's end: it is now mounted volatile (Linux 5.10; older kernels mount it as before). Publishing a layer synced the whole store's filesystem (syncfs, #272): a layer of up to 256 entries now fsyncs only its own files and directories, from 16 threads so the journal commits them together, and a larger one syncs the filesystem as before. A published entry is still never visible before its contents are on disk, and the job syncs the layers' directory once at its end. On a laptop with another process filling the same disk, a cached job's syncs went from 42 s to 5.5 s; alone, from 316 ms to 259 ms. A helper now reads its spec in one go, not a byte at a time. Native jobs also say where their time goes: the runner's log gets one line per native job with every phase of its set-up, of each sandbox, of each layer published and of its teardown, and with step debug "Set up job" and each step show theirs. .github/scripts/native-phases.sh turns those lines into per-phase medians; the end-to-end suite prints them after its native pass, and on razorbill journalctl -u zig-runner@bench.service | .github/scripts/native-phases.sh does.
• A registry token request whose kept-alive connection was reset is sent again, as a closed one already was, instead of failing the pull.
• A rootfs, a seed, an image, and an image's blobs are on disk before their names are, so a power loss cannot leave one empty under the name that says it is whole; a failed fsync in a publish is no longer lost when another file of the layer could not be opened.
• The first native job after a boot checks every stored layer's tree against what was published and evicts one a crash damaged. A large layer's publish reports a write error from since its job began even if another job's sync was told first, and a kept layer digest is on disk before its name and refused unless whole.