Trace-Topology nightly: 13 consecutive failures since 2026-08-08 — podman cannot write /run/user/1109, and the guard blames /dev/kvm instead

[avrabe]

The measurement

Trace-Topology Fixture Generation has failed 13 consecutive runs, every
night since 2026-08-08. Last success: run
31147914827
(2026-08-07, 1846d9a3). First failure of the streak:
31238093810
(2026-08-08). Most recent:
32328118359
(2026-08-20, 1998932b).

Sampled the first, a middle, and the most recent failure — byte-identical
cause in all three, so this is one regression rather than a drifting set:

level=error msg="set sticky bit on: chmod /run/user/1109/libpod: read-only file system"
##[error]Process completed with exit code 1.

The failure lands 0.03 s after the step starts. podman never launches a
container.

It is the host, not the workflow, and not the model

The runner is unchanged across the boundary — gh api on the job of the last
success and of the current failure both report:

runner=pulseengine-ci-01-9  group=hetzner-private  labels=self-hosted,linux,x64,podman

Same machine, same labels. What changed on 2026-08-08 is that /run/user/1109
— the runner user's XDG_RUNTIME_DIR — became read-only, so rootless
podman cannot create libpod inside it.

The workflow is exonerated by its own history rather than by assertion: the
last commit touching trace-fixtures.yml is 0ca6f68 on 2026-08-05, and
three nightlies ran green after it (2026-08-05 10:25, 08-06, 08-07) before
the streak began. So the file that was last edited three green runs earlier is
not what broke on 08-08.

This is also not #362 (uncommitted flake.lock) or #365 (missing qcow2
attribute) returning — those were both inside the flake; this one is upstream
of Nix entirely, since no container ever starts.

The diagnostic names the wrong cause

The step that fails is Require /dev/kvm openable by a Nix BUILD process, and
its failure branch prints ~20 lines of ::error about PODMAN_CAPS,
virtiofsd/capng_apply, --group-add keep-groups, and build-users-group,
ending with:

the probe log above says which of the two preconditions failed; read it
before this list.

There is no probe log. podman exited before the probe ran, so the guard
confidently attributes the failure to a capability/KVM-group problem that is
not what happened. Anyone triaging this from the log alone is sent to the wrong
subsystem — a guard whose error path describes a measurement it never took,
the same class this repo has been closing all through v0.36.0/v0.37.0
(REQ-GUARD-GATE-EVIDENCE-002).

Proposed fix — two parts, both needed

1. Stop depending on the host's session state. No workflow in this repo
   sets XDG_RUNTIME_DIR (grep -rn XDG_RUNTIME_DIR .github/workflows/ →
   nothing). Point podman's runtime state at a path the job owns, e.g.
   XDG_RUNTIME_DIR: ${{ runner.temp }}/podman-run (plus --root / --runroot
   if the storage dir has the same exposure). A nightly whose liveness depends
   on whether systemd happens to have a live user session for uid 1109 will
   keep dying this way.

2. Make the precondition step fail with its own cause. Assert
   podman info succeeds before the KVM probe, and emit that error. The
   current step cannot distinguish "podman did not start" from "kvm is not
   openable", and prints the second either way.

Not proposing to fix the host here — but if the intended fix is host-side
(loginctl enable-linger 1109, or remounting /run/user/1109 rw), part 2
still stands on its own: the streak was 13 nights long partly because the log
pointed away from the real cause.

Not claimed

That the fixture VM would otherwise be green. Once podman starts, the KVM
probe and the Nix VM build have to pass on their own merits, and they have
their own history (#362, #365). This issue is only about the 13-run streak
whose cause is /run/user/1109 and about the guard that mislabels it.

🤖 Generated with Claude Code