v0.29.0

v0.29.0 - 2026-11-09

Added

• Native backend: a job container's named volumes outlive the job as Docker's do, kept in the native store per repository and trust scope, filled from the image when new, read-only with :ro, and evicted like the cache volumes
• Native backend: a job container's host-path volumes, off by default, bound only from the directories RUNNER_NATIVE_HOST_PATHS allows (never for untrusted jobs), nosuid and nodev, read-only with :ro, idmapped through the volumes' id range; a path not allowed fails the job naming the setting
• The runner can pull the default job image already provisioned, built by a new "provisioned image" workflow for linux/amd64 and linux/arm64 from the runner's own provisioning script and pinned by digest, instead of running apt-get on every new machine and empty store (22-45 s, and Ubuntu's mirrors). A pull that fails provisions as before and "Set up job" says so; RUNNER_PROVISIONED_IMAGE=false never pulls it. Nothing is pinned yet, so nothing changes until a maintainer publishes the images and pins them (provisioned-pin.sh).
• Native backend: a job container's ports: are published from the host by a relay of the runner's (TCP and UDP), on 127.0.0.1 unless RUNNER_NATIVE_PORT_ADDRESSES allows other addresses, with a bare port given one the kernel picks and reported as docker port does; no port below 1024, none for untrusted jobs
• The nightly fuzz job has a zstd target that holds the runner's own zstd decoder to std's on every input (the same bytes or both an error, within a window limit, a heap limit and an output cap, and no hang), seeded with frames of every literals and sequences mode, and a test that flips each bit of the small seeds. It found three things std's decoder takes that libzstd refuses (an offset past the window, a Huffman tree that is not complete, a literal stream not used up), which the runner's decoder refuses with errors of their own; the decoder also now refuses a block over its window and a frame that is not the size its header says
• A Docker action whose image runs as a non-root USER runs on the native backend on a root runner: the workspace, home, file commands and event are the image's user's through an idmap of the job's layers that swaps it with the job's user, so what the action writes is the job's user's afterwards (#351)
• Native job containers honour more of options:: --hostname, --add-host, --shm-size, --tmpfs (nosuid and nodev, noexec unless asked), --memory (within the job's cgroup limit, never above it) and --init (#352). Security-sensitive flags (--privileged, --cap-add, --security-opt, --device, --network, --pid, --ipc, --userns) stay refused, naming the flag. A native job no longer logs the Dagger path's "options not supported here and ignored" warning for settings it carries out.

Changed

• Native image pulls unpack several times faster: zstd layers use a decoder of the runner's own (std's took 19 s for a 500 MB layer, now 1 s), a layer's inflating and hashing run on a thread of their own beside the unpacking, the layers' blobs are fetched ahead while the earlier ones unpack, and large files start their writeback as they are written so the store's syncfs has little left to wait for
• CI scripts stream their progress into the job log as they run (tlc.sh and spin.sh per configuration, zig-stress.sh per round, store-crash.sh every 25 crash points, the comparison scripts per phase), not only a summary at the end
• Valgrind in CI stops a run whose log has not grown for ten minutes (eight for a pull request's changed tests) and names the test that was running, instead of waiting for the job's limit
• The README is now a short overview; contributing (set-up, hooks, versioning, changelog) moved to CONTRIBUTING.md, the CI pipeline to docs/CI.md, and running the runner (trust, caches, deploying, memory) to docs/OPERATIONS.md
• A release's provenance bundle is attached as zig-runner.provenance.intoto.jsonl (was zig-runner.provenance.jsonl), the name OpenSSF Scorecard's Signed-Releases check recognises
• CI runs eviction's really-full-disk test (full-disk-test.sh) on every pull request that can affect the tests, instead of only by hand
• docs/native-backend.md says the published ports' 127/8 exemption assumes route_localnet is off on the host's bridges, and what a container can send when it is on
• A native job container's --tmpfs is refused above the runner's mounts and files (/github, /etc) and where an image's symlink leads onto them, and with no job cgroup (RUNNER_NATIVE_CGROUP=none) an unsized --tmpfs or more than 1 GiB of --tmpfs and --shm-size fails the job

Fixed

• Native backend: a job container's host-path volumes are refused on a mount without nosuid and nodev on the host, nested with another volume, on a rootless runner, or on a filesystem that cannot be idmapped unless RUNNER_NATIVE_HOST_PATHS_UNMAPPED=true; mount points are made before any bind and through no mount; each job is held to the list checked at start; the runner's executable, App key directory and env files are refused; and a refused source says nothing of what exists on the host
• Native backend: a job container's explicit host ports must be inside RUNNER_NATIVE_PORT_RANGE (49152-65535 by default) and never the runner's own database, engine or Docker port; ports published beyond loopback refuse peers of the runner's jobs and of the host's bridges; 127.0.0.1 is published on ::1 too; the relay connects without blocking, closes idle connections, lets a newcomer replace the quietest and caps new UDP clients a second
• Native backend: a job container's named volumes are refused for a job the runner does not vouch for and for a fork's pull request, an open waits for another job's filling of one only so long and not past a cancel, and the filling is stopped after 10 minutes or at the job's cancel
• Named volumes: a fork's cache growth no longer evicts a repository's named volume ahead of untrusted ones; repository-scope volumes rank a week later, as images rank an hour later (#428)
• Published ports (#428): the relay also refuses peers reached through a veth into another namespace or a tun/tap device (routed pods, VMs), keeps a UDP client's verdict for a second instead of reading the routes per datagram, and counts an engine or Docker host written without tcp:// as reserved; the range refusal names RUNNER_NATIVE_PORT_RANGE=1024-65535 for Docker's behaviour.
• A native job's Zig cache seed that an eviction took just as it was built is built again (up to three times) instead of leaving the job unseeded
• Eviction no longer overruns a job's end on a chain of directories with no files: a walk of a volume asks the clock at every directory, and a directory too deep to name is not walked a component at a time. A volume too big to walk in one pass is counted over several, resuming from the directories the last one left in its size file. A dead tree that will not go is left alone for an hour instead of costing a syncfs at every job's end (#357).
• The end to end suite reads a cancelled job's steps only once each has its conclusion, not when GitHub's step list still lags behind the finished job (#437)
• A host path (RUNNER_NATIVE_HOST_PATHS) that the runner's mount table also shows at another path without nosuid and nodev is refused at start and closed for each job; the docs say to mount the whole filesystem nosuid,nodev, since a self-bind covers only the runner's mount namespace and is lost at reboot
• A docker:// step that never runs (an if: that is false whatever the job's state) no longer decides whether the job runs natively: its image is not pulled, so one that cannot be pulled no longer moves the whole job to Dagger (#351)

v0.28.1

v0.28.1 - 2026-11-08

Fixed

• The runner no longer aborts with "integer overflow" at the first job end after a start, when the memory gate first writes what it learned (v0.28.0); each pool on a machine keeps its learned sizes in a file of its own, memory-gate-<RUNNER_NAME_PREFIX>

v0.28.0

v0.28.0 - 2026-11-08

Added

• A TLA+ model of the release pipeline as #408 has it (proofs/tla/ReleasePipeline.tla), checked by TLC in CI with its mutants: tags only checked commits, deploys only checked releases, waits for the current check names, and ends every wait
• RUNNER_PREWARM=before-listen (or --prewarm=before-listen) finishes the native store pre-warm before any runner registers or polls, so a machine added to a pool takes jobs only once warm; a failed warm, or one over 15 minutes, still lets the runners listen, and a stop ends the wait

Changed

• TLC runs in four parts side by side, each configuration given to a part by its expected time, so the model check takes about three minutes instead of twelve
• The analysis workflow splits its Coverage job into four parts of a quarter of the tests each, merged by a final Coverage job, and its Stress job into four jobs of 25 rounds, so each finishes in about a quarter of the time.
• The pre-push hook also runs the tests on overlayfs and ZLint when the runner changed, and TLC when the models or the release pipeline changed, so failures CI found first are found before a push
• TLC checks what its configurations claim, a little faster than it did: the store's locking model ordered a directory listing by the runners' names, which the runners' symmetry does not allow, so TLC left part of the model unexplored (StoreLocking-crash.cfg 6.5 million states where every orbit is 7.3 million); the order is now the lister's own and every orbit is explored. The release pipeline's safety is checked under a view that leaves out what no action reads again (2.0 million states to 156,000, the same violations found), and its liveness as two configurations of one property each
• StoreLocking's safety is model-checked with SPIN, on a Promela port of the TLA+ model that reaches exactly TLC's states, in a fifth of the time; its liveness stays TLC's

Fixed

• The release and the deploy wait for the check names in .github/checks-on-main, read from the commit they check out, and fail at once on one that never ran; CI checks those names and the rulesets' required checks are jobs that run
• The memory gate no longer charges a native pool's idle runners to the engine's cap (#414), counts only the job that has waited longest against small jobs (#415), keeps what jobs took across restarts (#407), and refuses to start, or lets one runner poll, when a single job could never fit (#406).
• The memory gate learns how long each job runs, and lets a job whose name ran 60 s at most each time start beside a heavy one waiting for room, so a burst of small jobs no longer queues at GitHub behind it (#415); the memory-gate file is now format v2 and a v1 file is ignored with a warning
• A stop during RUNNER_PREWARM=before-listen is no longer said as a finished warm with the runners about to register, when the stopped warm was done before the wait first looked

v0.27.0

v0.27.0 - 2026-11-07

Added

• A native pool warms its store when it starts: the default image is provisioned and the Zig cache seeds are built in the background, with or without an engine (RUNNER_ENGINE=none), so the first jobs on an empty store no longer wait for them. A failed warm is only said.
• A container: job's volumes: that are directories of the job (an anonymous /data, a named cache:/cache) run on the native backend: each is made in the base, root's, there for every step and gone with the job. Host paths, read-only volumes, a named volume a service also lists and odd targets still go to Dagger with the volume named, as does ports: (publishing to the host needs a network rule the runner does not have); the reasons say Dagger ignores them too.
• The engine container can be set with --engine-container, as RUNNER_ENGINE_CONTAINER does.

Changed

• The helpers loop.zig and loop/job.zig share (say, fail, failBody, mintToken, jsonString) moved to loop/exchange.zig, so the two files share little more than Runner, Cycle, Registering and handleJob (#148).
• The part of the dagger.zig overview about markers and reading the log moved to dagger/JobLog.zig, next to the code it describes (#148).
• The README links the SPEC, DESIGN and STUDY documents that the code comments cite, at the commit the runner was split from.
• Declarations in dagger.zig and dagger/ that other files use got doc comments, ten that no other file uses are private again, and docker.dockerRunFor is now docker.runFor (#148).
• The select that runs a function next to a timer and closes a result that arrives late, repeated in pg.Client.connect and feed.Socket.openWithin, is one helper, http.withinClosing, beside http.within (#148).
• The database store and the log sink share one record of their last failure and back-off, and one close-wait, instead of two copies.
• The job message and results-service code share one reader for a mapping's Key/Value entries and one wrapper that turns a failed try into a Result.
• The corpus comparison tools share unhex and the reference loader in runner/tools/corpus.zig.
• BENCHMARKS.md, THIRD_PARTY.md and CHANGELOG.md moved into docs/, with every link and reference to them updated; README.md stays at the root
• CI runs the native tests as root in two parts per architecture, side by side, and the arm64 release builds apart from its tests; the required checks are now gates over those jobs
• The required checks are the test jobs themselves (Tests on both architectures, Release builds (arm64) and the four parts of the native tests as root) instead of two gate jobs, which waited up to 14 minutes for a runner in a busy merge queue

Fixed

• A native step that is cancelled, times out or is stopped with the runner now gets GitHub's warning before it is killed: SIGINT to every process of the step, SIGTERM 7.5 s later, the kill 2.5 s after that, all inside the job's cancel limit, and the step's log says why it was stopped. A second cancel from GitHub can now shorten the limit.
• A native exec stopped during a cancel never counts as exiting 0, so a Dockerfile RUN cut short stores no layer, however its script ended. A step stopped is reported cancelled, or failed on a timeout, whatever it exited with. GitHub's cancel limit is taken as its runner takes it: at least 60 s, less 15 s.
• A pull request based on another branch no longer turns on auto-merge, which merged a stacked pull request as it opened, before CI ran.
• A native pool no longer holds small jobs behind a heavy one that is not there (#361): a runner polling reserves RUNNER_NATIVE_MEM_PER_JOB_MB instead of the heaviest job learned, and a job waits for room for its own estimate once GitHub has handed it out and named it, kept by the renewed lock, one at a time under the gate so a burst of heavy jobs starts only as far as there is room.
• The tests of loop/job.zig (the job's log spool, the cancel watch, the stop deadline) were never part of the test build; they run now.
• The README says automerge skips pull requests based on another branch, as automerge.yml does.
• Native eviction's floor is now tested on a really full disk (a tmpfs filled until writes fail with ENOSPC, in a bubblewrap mount namespace) as well as on simulated shortages.
• The unsupported-step-reference message names the types the runner supports, and inspect no longer crashes on an unexpected job message.
• The test of an abort that comes as the answer does no longer races a 10 ms timer, which a busy CI machine could pause past and fail

v0.26.0

v0.26.0 - 2026-11-07

Added

• Jobs that use hashFiles() run on the native backend. The runner reads the job's layers itself, as the next step sees them (whiteouts and opaque directories included), with no mount and no Dagger engine; a link is hashed as what it leads to inside the job's tree, never the host's, and the result is the same files, order and hash as on Dagger. A call is bounded (files visited, steps, memory shared by all jobs, GitHub's 120 seconds; RUNNER_NATIVE_HASHFILES_MEMORY_MB and RUNNER_NATIVE_HASHFILES_TIMEOUT_S) and stops when the job is cancelled.
• The systemd unit sets MemoryMax=90%, so a runner that runs away stops its own pool, which systemd restarts, rather than leaving the kernel to pick what to kill on the host. Existing installs need the setting added by hand: deploys do not replace the unit.
• Native cache volumes are evicted: past RUNNER_NATIVE_VOLUMES_MAX_MB (16 GiB), or with under RUNNER_NATIVE_MIN_FREE_PCT (10) of the disk free, those no job holds go, the longest unused first. A volume used today stays over the cap; one evicted is an empty directory to the next job.
• Native images, provisioned rootfs, Zig cache seeds and blobs are evicted: past RUNNER_NATIVE_IMAGES_MAX_MB (8 GiB) together, or under the disk floor, the blobs go first, then what no job holds, the longest unused first, a rootfs before its image. A job holds each until it ends. Something used today stays, so the base image is not evicted on a loop.
• RUNNER_ENGINE=none (with RUNNER_BACKEND=native) runs the runner with no Dagger engine: no pre-warm, no engine look in the memory gate, and a job native cannot run fails at once in Set up job, naming why, instead of waiting for an engine. deploy/compose.engineless.yaml drops the engine service. The default is unchanged.
• The native backend fetches a checkout submodules: (true the first level, recursive all, as GitHub does) in the job image, so these jobs no longer fall back to Dagger. The flags are part of the step key, so a checkout with other flags is never replayed.
• A container: job's options: run natively when they are --env, --user, --workdir or --health-*; any other flag is refused by name and the job runs on Dagger. ports: and volumes: still run on Dagger.
• An optional AppArmor profile (deploy/apparmor/) for Ubuntu 23.10 and later, which grants the runner binary userns so rootless native sandboxes work without lifting kernel.apparmor_restrict_unprivileged_userns; the refusal names it when that restriction stops the sandbox (#380)

Fixed

• A check fails CI and the pre-push hook when a file's tests are not in the test binary, so a file left off main.zig's test block cannot pass with its tests never run
• A runner held at the memory gate looks again as soon as a job starts, is named or ends, instead of at its next 5 s check, so a burst of short jobs no longer waits out the check once per job.
• The end-to-end suite says why a job has no log, with what the runner reported of its log uploads, and does not wait for one the runner says failed
• A rerun of a job that installs a tool into the tool cache (Install Zig) replays that step on the native backend, as on Dagger. "Set up job" checks the tool cache's trees in the runner itself, bounded and confined to the volume, and a step whose record names a tree that is still the same is no longer run again.
• Runners starting together on one host no longer send a native job to Dagger: they take turns making the NAT rules, and a rule batch that another runner overtook is tried again after a pause, up to twelve times. A job sent to Dagger that cannot start it now fails saying why it was sent.
• A native image pull whose token request fails after its resets says how many times it sent it, and the stale-connection test says how many requests the registry read.
• A native checkout of a commit with fetch-tags is no longer replayed with tags that have since moved: the remote's tags are part of its key
• A job's log and its steps' logs are uploaded again after a transient failure (DNS, connection, timeout, 5xx, 429), with a bounded backoff, instead of being lost to one network blip
• The native store's TLA+ model follows the publish as #309 left it, the machine's crash loses renames no syncLayers covered, and the jobs after one run through checkOncePerBoot; every configuration holds, with three new mutants.
• A native layer counts its record and its own files against RUNNER_NATIVE_STORE_MAX_MB, not only its tree, so steps that change nothing no longer grow the store for ever under a cap that saw 0 bytes.
• Three tests that failed with nothing wrong: a rename race that never ended under Valgrind, an output keys check tripped by a random name, and a fuzz check tripped by a repeated JSON key
• Under the native disk floor layers go first, and every native job sent to Dagger logs why.
• A native Zig cache seed built again is a new key, and a rootfs in the store keeps its fingerprint while a sibling sharing its files is evicted. Layers on such a rootfs miss once at the upgrade.
• The nightly suppressed-errors check accepts the best-effort sites hashFiles and the tool trees added
• A task's cancel that lands in hashFiles' wait for its turn, or in the tool trees' upkeep of their lists, is no longer lost
• A job no longer fails when the Dagger engine refuses to give an image's digest after a pull that worked ("Image reference can only be retrieved immediately after the Container.From call"): the digest is only a log line, so it is asked for on its own and its Digest line is left out when refused. A pull that fails still fails the job or service, and the pre-warm falls back to the pinned image name.
• On a filesystem where a read lease does not see a writable mapping, such as overlayfs, the tool trees' check reads every file again instead of trusting its list
• Tool-tree checks and hashFiles() now count their memory in one number, so the runner has a single limit for what a job's checks and hashFiles() hold at once
• The weekly hashFiles() comparison with @actions/glob now covers the native backend's hashFiles() too, not only glob.zig
• Native eviction's removal of a dead tree counts going down a directory against its budget, so a chain of directories with no files no longer overruns it
• CI tests on overlayfs and checks ZLint's suppressed errors on every pull request that changes runner code, and runs Valgrind on the tests a pull request adds or edits, stopping a Valgrind run that does not finish and naming its test
• The Dagger path warns in "Set up job" about a job container's options:, ports: and volumes:, which it does not apply, naming each value, instead of dropping them silently (#368).
• Native jobs are bounded together: a limit on their parent cgroup leaves the machine a third of its memory, jobs do not swap, and a job that cannot be limited does not run natively unless RUNNER_NATIVE_CGROUP=none accepts the risk
• With RUNNER_ENGINE=cli, a GitHub cancel, a stop or a step timeout ends the running step (#135). The query to the dagger run session is dropped and its connection closed, as the direct session does, so the session stays up for the always() steps that follow; before, the step ran to its end. An abort that lands as the answer arrives is not lost.
• A job container's options: -e no longer beats the container's own env: (GitHub puts env: last), -e HOME has no effect, a --user on an image without a writable /__w works because the base makes the workspace as root and hands it over, a --user the image does not know fails the job instead of falling back to Dagger as root, and the note for a refused flag says Dagger drops it
• RUNNER_ENGINE=none now checks at start-up that native steps can run and refuses to start with the reason, instead of failing every job; it rejects RUNNER_DAGGER_NESTING; and deploy/compose.engineless.yaml gives the runner the privileges, host network, private cgroup, store volume and /run/zig-runner the native backend needs
• Tool-tree checks are limited by what checks hold (128 MiB together), not by what hashFiles() calls hold too, so two or three large hashFiles() calls no longer leave every other job's trees out; the two still share one visible number (the earlier note's single limit was wrong). The eviction descents counter wraps instead of panicking a ReleaseSafe runner after 2^32 descents. A native checkout with submodules and tags is not replayed, as a submodule's tags can move
• A native job's OOM kill now takes every process of its steps together (memory.oom.group on a steps child of the job's cgroup), not only the largest one, so a step no longer carries on in a damaged state; services and the ports' relay run in a sibling cgroup without it and survive a step's OOM for the cleanup steps. Job processes also get oom_score_adj 500, as GitHub's runner does (PIPELINE_JOB_OOMSCOREADJ overrides it, never below the runner's own), so a machine-wide OOM prefers a job to the runner. Both are best effort.
• A pull request merged by the automerge app now closes the issues it links with Closes #N, which GitHub left open.

Security

• A symlink a job swaps in at a nested cache volume is refused rather than bound, and a root runner's overlays turn off redirects and metacopy whatever the kernel defaults to
• A symlink a job plants anywhere below the volumes root, not only at a nested volume's last name, is refused rather than bound
• Native steps have their user's supplementary groups, as on Dagger, and hashFiles counts that same list, so editing /etc/group gains a step nothing
• On a native backend, hashFiles() refuses a file the step's user could not read, as under Dagger, such as through a link to /etc/shadow

v0.25.0

v0.25.0 - 2026-11-06

Added

• The native layer store is crash-tested. In CI, Store.publish, the once-per-boot check, oci.Store.ensure and volumes.adoptSeed run on ext4 over dm-log-writes, the disk is put back to every flush, FUA and mark of the log and mounted, and no layer open accepts differs from what was published; and a publish on a disk that fails writes (dm-flakey) must fail, in syncTree, in syncfs and through the job's descriptor, rather than publish.
• The tests run in as many processes as the machine has cores, in zig build test, CI and the pre-push hook alike (runner/tools/test_shards.zig; ZIG_TEST_SHARDS=1 for one). The tests that use fixed ports on the host, and as root all the native backend's, still run one at a time.

Fixed

• A native job near its memory.high gets more while the host has memory to spare. A first zig build test was held at the 1 GB a job with no history gets, and spent over an hour in swap with 4 GB free, where the Dagger backend took 2 minutes.
• A native publish commits a layer in one journal commit, not two or three. The layer's data is written back first and its meta written before the fsyncs, so a small layer's publish takes about half the time.
• The end-to-end suite tolerates GitHub's API lag: it re-reads jobs and runs that are completed but still list no conclusion, starts the cancel when the job has plainly been running for 30 s even if the step list does not show Hog (the checks still need Hog cancelled and its output in the log), and bounds the wait for job logs GitHub does not serve (one 60 s budget for the suite, and a skipped job is asked once).
• With RUNNER_ZIG_LOCAL_CACHE, an unchanged commit no longer recompiles. Zig's local cache names the builtin.zig files it generated in the global cache's b/, which is each job's own, so every job found them gone and compiled the tests and the runner again (9 s and 8 s on the laptop, cached now). b/ has a volume of its own beside the local cache's, in the same repo scope, filled from the seed's so that the seed's build runner still hits; on the Dagger and the native backends.
• The allocation-failure tests rerun on a faster allocator and count what is left themselves, so they take half as long with the same checks: a quarter off the test run of every build and push.
• Running out of memory while reading the Results service's answer to a log or step summary upload is reported as that, not as "no logs_url in response" (or "no summary_url"): the lookup swallowed the allocation failure as a missing key.
• A rerun of the same commit replays its steps again. RUNNER_TOOL_TREES (on by default) reached --exec-job but never the jobs the runner's loop runs, so "Set up job" did not check the tool cache's trees, and a replayed step that put a directory of the tool cache on the PATH (Install Zig) ran again, with every step after it: a rerun took as long as the first run, 41-48 s against 2 s on the laptop.
• A native publish of a checkout no longer waits for everything else being written to the disk. Layers of up to 512 entries sync only their own files, up from 256.
• The end-to-end suite checks the replay of cache on its third attempt. With the tool cache's trees checked (RUNNER_TOOL_TREES), the tree list is in every step's key, so a tree installed by the run's toolcache job between two attempts makes the next one run its steps again, once; the second attempt now only has to succeed.
• Native Dockerfile actions: a build's stored layers are keyed on what the base image is, not only its tag, so a tag that moved builds again instead of stacking layers from the old image on the new one. Build keys also keep a root and a rootless runner apart and hash every field length-prefixed. The store format is now 5, so existing layers are rebuilt once.
• The native services test asks for the redis image CI fetches, and a change to those images runs the tests. Renovate moved the images to redis 8 on a change that skipped the tests, which left every later test run failing.
• A native job no longer fails at random unpacking an image with a .. symlink while another job publishes a layer. The kernel refuses such a lookup with EAGAIN when any rename on the machine races it, and openIn now tries again.
• A runner that has been up for a while no longer loses GitHub when a certificate rotates. A certificate issued after the runner started failed every new connection until a restart: std reports any failed handshake as TlsInitializationFailed, which the check that replaces the client did not match. The client is now replaced after such a failure, at most once in 15 minutes, and once it is a day old, so it also stops accepting a certificate that expired since it started; and github/http.zigs tests, which the build never listed, now run.

Security

• The inputs #107 listed as untracked are pinned and tracked: the JavaScript runtime images for actions (oven/bun, node) by multi-arch index digest and zizmor by version and file hashes (.github/zizmor-requirements.txt, installed by .github/scripts/zizmor.sh with --require-hashes). Renovate moves both, and leaves a Node or Bun major to be moved by hand.

v0.24.4

v0.24.4 - 2026-11-04

Added

• Mutation testing of the security-critical modules (secret masking, workflow commands, expressions, the native store, minisign): runner/tools/mutate.py changes a module's code one thing at a time (a comparison flipped, and for or, a return error.X guard taken away, a constant off by one, a statement deleted), runs that module's tests on each change, and reports every change they let through by file and line. The mutation workflow runs it for one module when asked for, over four of GitHub's runners, with the report as an artifact; it is too slow for a pull request. New tests kill the most important survivors of the first run: in masking (a lone secret, a one-byte one, a line of exactly four bytes, URL-encoded forms, bytes past ASCII, the stop-commands token), the native store (collect keeping what it should and evicting least recently used first, a changed record, half-built layers, record limits), workflow commands (an escape cut short at the end of the data, which a mutant read past), minisign (the key's algorithm tag, the trusted comment's prefix and limit), and if: conditions after a failure.
• Line coverage of the runner's tests with kcov (Ubuntu's package, pinned), nightly and on every pull request that changes runner/src, as analysis's "Coverage" job: the rootless tests, the native backend's root pass and the runner binary they start for their sandboxes, merged into an HTML report with Cobertura XML and JSON (the coverage artifact), and a job summary with the total, each directory and the least-covered security-relevant files. It reports without failing; COVERAGE_MIN sets a floor later. Nothing is sent to a coverage service.
• Store stress runs on a pull request that changes the native store, its layers, or the locks and mounts around them (the new store scope), not only nightly.

Fixed

• The native layer store refuses to publish a layer unless its key's lock is held, so a caller that forgets it (as Dockerfile builds once did) fails at once instead of racing another runner's collect. A new stress test runs six processes against one native store under a small cap for 30 seconds (rootless and as root, on x86_64 and arm64; ten minutes with eight processes in the nightly analysis workflow), each running jobs and Dockerfile builds on overlapping keys, and checks that every replay matches the live run that stored it, no job sees an evicted or half-published layer, every entry stays whole, no key runs in two jobs at once, no fd or lock leaks and no warning is logged, and a last collect leaves no temporary entry and the store within its cap. It found the Dockerfile build lock race independently of the store's TLA+ model.
• A native job's directory is no longer removed by another runner while its own runner is removing it: the job's lock file went early in that removal, and another runner sharing the store, starting a job, made a new lock in its place, found it free, took the directory for a dead runner's and removed it at the same time (logging AccessDenied when the two collided). A job's directory is now removed with its lock last, and a runner's sweep opens a lock without making one; a directory a dead runner really left is removed with a warning. Found by the store's stress test in its first ten-minute nightly run.
• The native layer store counts a layer's bytes once it is on disk: it measured the tree before its syncfs, while the kernel could still be writing back a file just written, and counted a block short now and then (seen on x86_64 and arm64, rootless and as root), so the store could hold a little more than its cap. The tree itself was always whole. And a Dockerfile build looks its layer up under the key's lock before publishing, as before the store's TLA+ model work: publishing first, a collect could evict the layer another builder stored between the refused publish and the lookup, and the build failed for nothing. The model now checks that no build is lost so (NoBuildLost), with the publish-first order as a mutant it must reject.
• A native job cancelled while a step waited for its key's lock (another job running the same step) or for a service to be ready now stops: both waits swallowed the cancel, which std.Io delivers once, and went on until the other job finished or the service's readiness limit passed. ZLint's suppressed-errors rule, which ci cannot run until ZLint applies line-level disables, now runs nightly (analysis's "ZLint (suppressed-errors)") against a baseline of the 119 sites read and meant, matched by file and a fingerprint of the line, and fails on a new one. CI's Workflows job also runs hadolint on the Dockerfile (whose findings are fixed: one RUN for the certs stage, pipefail for the build stage's; unversioned apk packages kept, with why) and shfmt on every shell script, both pinned and checked against their checksums, with Renovate following them.
• Native steps no longer wait for every other write to the disk they run on, which cost native jobs seconds on razorbill's SD card (toolcache 26 s and cache 20 s against Dagger's 6 s and 8 s), most of all while another job filled its tool cache. Each step's overlay synced its whole upper filesystem when it was unmounted, at every step's end: it is now mounted volatile (Linux 5.10; older kernels mount it as before). Publishing a layer synced the whole store's filesystem (syncfs, #272): a layer of up to 256 entries now fsyncs only its own files and directories, from 16 threads so the journal commits them together, and a larger one syncs the filesystem as before. A published entry is still never visible before its contents are on disk, and the job syncs the layers' directory once at its end. On a laptop with another process filling the same disk, a cached job's syncs went from 42 s to 5.5 s; alone, from 316 ms to 259 ms. A helper now reads its spec in one go, not a byte at a time. Native jobs also say where their time goes: the runner's log gets one line per native job with every phase of its set-up, of each sandbox, of each layer published and of its teardown, and with step debug "Set up job" and each step show theirs. .github/scripts/native-phases.sh turns those lines into per-phase medians; the end-to-end suite prints them after its native pass, and on razorbill journalctl -u zig-runner@bench.service | .github/scripts/native-phases.sh does.
• A registry token request whose kept-alive connection was reset is sent again, as a closed one already was, instead of failing the pull.
• A rootfs, a seed, an image, and an image's blobs are on disk before their names are, so a power loss cannot leave one empty under the name that says it is whole; a failed fsync in a publish is no longer lost when another file of the layer could not be opened.
• The first native job after a boot checks every stored layer's tree against what was published and evicts one a crash damaged. A large layer's publish reports a write error from since its job began even if another job's sync was told first, and a kept layer digest is on disk before its name and refused unless whole.

v0.24.3

v0.24.3 - 2026-11-02

Added

• A nightly "Stress" job in the analysis workflow: the runner's concurrency-sensitive tests (cancels, deadlines and timeouts, the memory gate, the native store's locks, pg's connect and the log writer, the broker's long-poll, log uploads) run 100 times over under stress-ng's CPU and memory load, with the database, failing on the first round that fails, its output an artifact. -Dtest-filter=upload now finds the log upload tests, and pg's test of a server that never answers no longer fails under Valgrind (its connect to the answering server had 200 ms). rr and Valgrind's Helgrind and DRD were tried and left out: GitHub's runner has no performance counters for rr, and Helgrind and DRD cannot see Zig's futex-based locks.
• More race detection. The nightly analysis workflow runs the native backend's tests under ThreadSanitizer on x86_64 and arm64, rootless and as root with the service images ("ThreadSanitizer, native"), with the runner binary they start for each step (zig-runner native-step) built with it too, as ZIG_TEST_SANITIZE_THREAD=1 now builds it; and the end-to-end suite on both backends with a runner built under ThreadSanitizer ("ThreadSanitizer, end to end", e2e.sh build BIN tsan), failing on any report in the runner's log or the jobs' (e2e.sh races). TSan's runtime needs glibc's dynamic loader, so these builds are dynamically linked PIEs: the release's static PIE crashes at startup with it. "Go tools" runs go test -race and a race build of each module under runner/tools, and runs rsadiff, the one tool with goroutines of its own, with that build.

Fixed

• The native layer store's locking is model-checked with TLC (proofs/tla/StoreLocking.tla, run by the new store-model workflow), and the model found two races, both fixed. A Dockerfile build's layer was published without its key's lock, so another runner's collect could take the half-published entry for a dead runner's and remove it while it was renamed into place, leaving a published layer with a half-deleted tree or no lock file; layers.zig now publishes under the key's lock, as a step does. And an eviction that opened an entry's lock file just before another runner evicted that entry would take the lock on the unlinked file and rename away the entry published under the key since, from under the job holding it; evict now checks its lock is on the file the path names, as open and lockKey do.
• A native runner whose binary a deployment replaced while it kept running (an instance not in zig-runner-deploy's UNITS, as razorbill's zig-runner@bench.service was) now goes on running native jobs: it took its own path from /proc/self/exe for each job, which then reads "/usr/local/bin/zig-runner (deleted)", and every job fell back to Dagger with "could not make the job's sandbox: FileNotFound" when its network helper could not be started. Its helpers now run /proc//exe, the binary it started from, until it restarts, and "Set up job" says so; a rootless runner that starts its helpers under unshare, which cannot run that, sends its jobs to Dagger saying why. Every native set-up failure now says what failed and on which path ("starting the job's network helper /usr/local/bin/zig-runner (deleted): FileNotFound"), not the error's name alone. A test replaces a running runner's binary as the deployment does and runs a native job on it, in both native passes.
• A problem matcher's pattern with a lookbehind no longer crashes the runner ("integer overflow") on a log line that is not valid UTF-8: a backreference compared bytes, so a lone 0xC3 matched the first byte of "À" and ended inside it, and a step back from there ((?=.(.))\1(?<=.) on C3 80 C3) went past the start of the line. Backreferences now compare whole characters, as .NET does, and a step back from inside a character takes the byte before as a character of its own, never more than the text before it. Found by the nightly fuzzer; its input is a seed now, the case is in the .NET comparison's fixed cases, and tests cover lookbehinds and backreferences (folded or not) that do not fit before or after where they are tried.

Security

• A new security workflow scans for secrets and checks dependencies. "Secrets" runs gitleaks (pinned by version and checksum, .github/scripts/install-gitleaks.sh) over each pull request's own commits and weekly over main's whole history, failing on any finding and never printing the value; the one finding in the history so far, a test's made-up layer key in runner/src/native/layers.zig, is allowed in .gitleaks.toml, narrowly and with why, and gitleaks:allow comments are ignored. "Dependency review" fails a pull request that brings a dependency with a known vulnerability of moderate severity or worse, or a license outside the permissive ones (MIT, Apache-2.0, BSD, ISC, Zlib, Unicode), as GitHub's dependency graph sees them: the Go tools' modules and the workflows' actions, not runner/build.zig.zon or pinned images. Renovate follows gitleaks' version.
• Fuzz targets and property checks for the text paths that guard secrets and a job's environment, run nightly by the analysis workflow, and the bugs they and their review found, each fixed: a GITHUB_OUTPUT or step summary value holding ::zig-file-command::env lines was read as the step's GITHUB_ENV (setting BASH_ENV ran code in later steps), now each file command's part carries its length; a heredoc value's EOF\r line ended a value delimited by EOF, so the lines after it set more variables, now file commands are read exactly as GitHub's runner reads them on Linux (a CRLF file's values keep their \r); overlapping secrets left most of one shown (***cdef), now they are masked as one span as GitHub masks them; secrets were not masked in base64 at an offset (base64(user:password)) or the other forms GitHub's runner masks (command line, expression and XML escapes, inside double quotes, PowerShell's halves); a secret across the cut of a line longer than 1 MiB was shown half on each line, and a long ::add-mask:: line was cut and its tail shown; and the engine's HTTP/2 connection kept CONTINUATION frames without end (a memory flood) and read frames past the 16 KiB a peer may send.

v0.24.2

v0.24.2 - 2026-11-02

Fixed

• Native jobs in an image without /etc/resolv.conf or /etc/hosts (Alpine's, and most of Docker Hub's but Debian's and Ubuntu's) now get the job's resolver and hosts file: a container: job, a Docker action, its Dockerfile's RUN and a service in such an image had no DNS, since the binds were made only over a file the image has. A layer of the job's below every image holds empty files at both paths, so the binds always have a mount point and no layer of the job's keeps one. A root test now checks that the built-in checkout's and an action's fetch resolve through the job's resolver as root and as the runner user with a Zig seed below (razorbill's toolcache job), and a Docker action in an image without a resolver of its own; the services test asks for postgres 18, as its image script now provides.
• The memory gate no longer holds native jobs back for memory they never use: a native job is learned from its own cgroup under its name, apart from the same job on Dagger, and one not learned yet reserves RUNNER_NATIVE_MEM_PER_JOB_MB (256 MB) instead of the engine's 1.5 GB, which held e2e jobs of a few MB on razorbill's native runners for up to 150 s. Each native job's cgroup gets a memory.high from that estimate (twice it, plus 512 MB), so a job that outgrows it is slowed there rather than taking its neighbours' memory; memory.max stays the hard cap. A job a native pool sends to Dagger is learned on the engine, and a Dagger job is no longer taken as running beside a native one.

v0.24.1

v0.24.1 - 2026-11-02

Added

• Zig-specific analysis of the runner's own code, which CodeQL and Semgrep cannot read: ZLint (pinned, checksum-checked) on every pull request ("Lint (ZLint)", with the rules that find bugs, runner/zlint.json); and a nightly analysis workflow that fuzzes each parser in runner/src/fuzz.zig with Zig's coverage-guided fuzzer, now including an image layer's tar, PAX and paths applied beside a directory it must never reach, image references and registry challenges, manifests and minisign signatures (each corpus kept in the Actions cache; a crash, failed check, leak or hang fails the job named after its target, with the input as an artifact), and runs every test under Valgrind's memcheck. zig build test runs the fuzz targets over their seed corpus every time; zig build test -Dllvm --fuzz fuzzes locally.

Fixed

• What ZLint's first run and the new fuzzing found: the database writer went on writing its whole queue after close gave up waiting for it (the cancel was swallowed), so a slow database held the runner's exit for a statement timeout per queued row; a runner held at the memory gate, or backing off after a failed registration, ignored the pool's cancel when another runner could not start, so the runner did not exit; and a workflow command the runner had no memory to carry out (set-output, save-state, set-env, add-path, add-mask, stop-commands) was dropped silently, where it now fails the step. hashFiles() with a pattern whose implicit descendants GitHub refuses (hashFiles('\.'), found by the new fuzzing) crashed the runner, every job on it included; it now fails as GitHub's does ("Relative pathing '.' and '..' is not allowed"). Unused declarations removed.
• The native backend on its first real host (razorbill: a Raspberry Pi with Tailscale): a root runner's jobs had no DNS when the host's resolver is not behind its default route (Tailscale's MagicDNS, 100.100.100.100 through tailscale0), so provisioning failed (Unable to locate package build-essential). Each job now gets a resolv.conf of its own, made from the host's as Docker makes a container's (no loopback or IPv6 nameservers, search domains and options kept, systemd-resolved's upstreams behind its stub, Google's public resolvers when nothing else is left, which "Set up job" then says), and the jobs' firewall lets UDP and TCP to port 53 of those nameservers out of whichever interface the host routes them by, and nothing else there; the rules follow the host's resolver when it changes. Native image pulls failed with HttpConnectionClosing once the process's HTTP client kept a connection Docker Hub had closed after a minute idle: a request on such a connection is now sent again on another, a bounded number of times. And a native set-up that fails before any step (the pull, provisioning, the base, the job's store, cgroup, slot or network) now sends the job to Dagger with the reason in "Set up job", as other unsupported cases do, instead of failing it; a cancelled job does not, and the memory gate measures the job on the engine again.
• The nightly OCI layer fuzz job runs 1M inputs instead of 3M: it took 27 of its 30 minutes as its kept corpus grew, and past 30 minutes zig-fuzz.sh reports a hang.

Security

• The release's binaries are hardened and gated: both are now static position-independent executables (-fPIE, and runner/build.zig's new pie option, on by default, so zig build, the image and the release still make the same binary), which the kernel loads at a random address instead of the fixed 0x1000000, with full RELRO, beside the non-executable stack and W^X segments they had. checksec 3.2 (slimm609/checksec, pinned by version and checksum, .github/scripts/checksec.sh) fails the release, and on pull requests "Release builds" and "Binaries (check)", on any binary without PIE, full RELRO, NX or W^X. A new check, "Release builds and tests (arm64)", builds the release on GitHub's arm64 runner, runs the aarch64 binary and the tests natively there (rootless and as root), so razorbill is no longer the first machine to execute an aarch64 build. PIE costs 0-3% on the runner's hot paths on the laptop (.github/scripts/hotbench.sh: expressions +2.9%, the hashFiles() glob +2.1%, YAML 4.2% faster) and nothing on the test suite (BENCHMARKS.md). Intel CET is not available: Zig 0.17 has no -fcf-protection for Zig code.