What happened
On 2026-09-04, an x86 gate lane building the #737 DF-preempt-oracle fix
scored another lane's serial as its own, because both lanes ran on
beast -> Incus container breenix-x86 and both invoked a gate script that
hardcodes an absolute /tmp/breenix_* output directory. From
docs/planning/green-program/nic-bus/737-DF-ORACLE-2026-09-04.md:
Because two lanes share that container and
docker/qemu/run-x86-boot-tests.sh hardcodes /tmp/breenix_x86_boot_tests_$i,
the first attempt of this slot scored another lane's serial: both scripts
rm -rf'd and recreated the same directory, and the surviving files were the
other clone's. Each run reported below therefore ran under unshare -m with
/tmp bind-mounted to a per-clone directory, which is why each boot's serial
is provably its own. That collision is a real defect in the shared harness
and is disclosed here rather than worked around silently.
The collision was caught only because the round checked the serial's content
against the boot image's own launch strings, not because the harness itself
noticed:
1 of 1 collided run identified by content, not by timing: the serial that
first attempt scored contains 15 RING3_SMOKE: creating lines and 0 for
df_preempt_oracle, while the boot image that run named contains the string
RING3_SMOKE: creating df_preempt_oracle (grep -ao over the image, 16
distinct launch strings). The other lane's QEMU command line, read from
/proc, names /root/breenix-787.
So the #737 lane's script rm -rf'd + recreated /tmp/breenix_x86_boot_tests_$i,
the #787 lane (clone /root/breenix-787) did the same to the identical path
at roughly the same time, and the #737 lane's poll loop read back whichever
lane's QEMU happened to still be writing that directory. Neither script's path
construction, rm -rf, nor verdict read-back (docker/qemu/run-x86-boot-tests.sh,
in full) checks which lane's clone or build produced the directory it is about
to read — that check does not exist anywhere in the file.
The follow-up round (docs/planning/green-program/nic-bus/737-FIX-2026-09-04.md)
names the fix as a per-run mitigation, not a structural one:
The /tmp collision hazard is mitigated per-run, not fixed.
docker/qemu/run-x86-boot-tests.sh and docker/qemu/run-boot-parallel.sh
still hardcode shared /tmp paths. 6 of 6 boots in this slot were isolated
with unshare -m plus a per-clone bind mount, which makes each serial here
provably its own; it does not help the next lane that forgets.
Which scripts, and the exact lines
$ grep -n "/tmp/breenix" docker/qemu/*.sh scripts/*.sh
The two sites this incident actually hit:
docker/qemu/run-x86-boot-tests.sh:355: OUTPUT_DIR="/tmp/breenix_x86_boot_tests_$i"
docker/qemu/run-boot-parallel.sh:75: OUTPUT_DIR="/tmp/breenix_boot_$i"
docker/qemu/run-boot-parallel.sh:134: OUTPUT_DIR="/tmp/breenix_boot_$i"
(run-boot-parallel.sh reconstructs the same /tmp/breenix_boot_$i string a
second time in its wait/verdict loop at line 134 rather than reusing the
variable set at line 75, which is its own small latent bug: the loop between
those two lines (docker/qemu/run-boot-parallel.sh lines 75-134) contains no
check that the directory still belongs to the process that created it.)
The full grep above (quoted in full at the top of this section) returns 50
matching lines across docker/qemu/*.sh and scripts/*.sh with the same
/tmp/breenix_* pattern (e.g. /tmp/breenix_aarch64_kthread_$i,
/tmp/breenix_kthread_$i, /tmp/breenix_gate_$i,
/tmp/breenix-parallels-serial.log). This issue is scoped to the two x86
boot-count scripts named above, the two that actually collided in production;
the remaining matches are candidates for the same fix but are not proposed
here.
Mitigation used today (works, but is opt-in and per-run)
unshare -m
# with /tmp bind-mounted to <clone>/gate-tmp/tmp
This isolates one lane's /tmp namespace from other processes on the host
via a private mount namespace, which is why the six boots reported in
737-FIX-2026-09-04.md are "provably its own." It is opt-in: whoever launches
the gate has to remember to invoke it each time. docker/qemu/run-x86-boot-tests.sh
and docker/qemu/run-boot-parallel.sh are unchanged by it — the unshare -m
wrapper is external to both scripts — so a lane launched without it can
reproduce today's collision.
Proposed fix
Add a BREENIX_GATE_TMP env var as the base directory for gate output,
defaulting to /tmp so a caller that does not set it (the current behavior of
both scripts, per the grep above) is byte-for-byte unaffected:
# top of docker/qemu/run-x86-boot-tests.sh and docker/qemu/run-boot-parallel.sh
BREENIX_GATE_TMP="${BREENIX_GATE_TMP:-/tmp}"
...
OUTPUT_DIR="$BREENIX_GATE_TMP/breenix_x86_boot_tests_$i" # run-x86-boot-tests.sh
OUTPUT_DIR="$BREENIX_GATE_TMP/breenix_boot_$i" # run-boot-parallel.sh (both sites, from one variable)A concurrent-lane launcher (or the unshare -m wrapper used today) then sets
BREENIX_GATE_TMP to a per-clone path instead of relying on mount-namespace
isolation nobody is required to remember. Any other script that reads these
same output directories downstream (verdict/tally readers) needs the same env
var threaded through, or it will look in /tmp/... while the boot itself
wrote to $BREENIX_GATE_TMP/....
The R18 lesson
A gate's PASS verdict was read off a directory, not off the lane's own
build. The fix above prevents the directory collision, but the harness should
also not trust "this directory contains a passing tally" on faith — the
collision here was only caught because someone diffed the serial's printed
launch strings against the boot image's own embedded strings after the fact.
A verdict reader should tie a PASS to the lane's own build hash (e.g. embed or
check the kernel ELF hash / a per-run token in the serial output it just
produced) so a future collision fails loudly instead of silently scoring the
wrong kernel as green.
Sources
docs/planning/green-program/nic-bus/737-DF-ORACLE-2026-09-04.md (section 2, the collision + how it was caught)docs/planning/green-program/nic-bus/737-FIX-2026-09-04.md (section 8, "mitigated per-run, not fixed")docs/planning/green-program/nic-bus/serials/737-df-fix-2026-09-04/hashes.txt (the unshare -m + per-clone bind-mount record)
What happened
On 2026-09-04, an x86 gate lane building the
#737DF-preempt-oracle fixscored another lane's serial as its own, because both lanes ran on
beast-> Incus containerbreenix-x86and both invoked a gate script thathardcodes an absolute
/tmp/breenix_*output directory. Fromdocs/planning/green-program/nic-bus/737-DF-ORACLE-2026-09-04.md:The collision was caught only because the round checked the serial's content
against the boot image's own launch strings, not because the harness itself
noticed:
So the
#737lane's scriptrm -rf'd + recreated/tmp/breenix_x86_boot_tests_$i,the
#787lane (clone/root/breenix-787) did the same to the identical pathat roughly the same time, and the
#737lane's poll loop read back whicheverlane's QEMU happened to still be writing that directory. Neither script's path
construction,
rm -rf, nor verdict read-back (docker/qemu/run-x86-boot-tests.sh,in full) checks which lane's clone or build produced the directory it is about
to read — that check does not exist anywhere in the file.
The follow-up round (
docs/planning/green-program/nic-bus/737-FIX-2026-09-04.md)names the fix as a per-run mitigation, not a structural one:
Which scripts, and the exact lines
The two sites this incident actually hit:
(
run-boot-parallel.shreconstructs the same/tmp/breenix_boot_$istring asecond time in its wait/verdict loop at line 134 rather than reusing the
variable set at line 75, which is its own small latent bug: the loop between
those two lines (
docker/qemu/run-boot-parallel.shlines 75-134) contains nocheck that the directory still belongs to the process that created it.)
The full grep above (quoted in full at the top of this section) returns 50
matching lines across
docker/qemu/*.shandscripts/*.shwith the same/tmp/breenix_*pattern (e.g./tmp/breenix_aarch64_kthread_$i,/tmp/breenix_kthread_$i,/tmp/breenix_gate_$i,/tmp/breenix-parallels-serial.log). This issue is scoped to the two x86boot-count scripts named above, the two that actually collided in production;
the remaining matches are candidates for the same fix but are not proposed
here.
Mitigation used today (works, but is opt-in and per-run)
This isolates one lane's
/tmpnamespace from other processes on the hostvia a private mount namespace, which is why the six boots reported in
737-FIX-2026-09-04.mdare "provably its own." It is opt-in: whoever launchesthe gate has to remember to invoke it each time.
docker/qemu/run-x86-boot-tests.shand
docker/qemu/run-boot-parallel.share unchanged by it — theunshare -mwrapper is external to both scripts — so a lane launched without it can
reproduce today's collision.
Proposed fix
Add a
BREENIX_GATE_TMPenv var as the base directory for gate output,defaulting to
/tmpso a caller that does not set it (the current behavior ofboth scripts, per the grep above) is byte-for-byte unaffected:
A concurrent-lane launcher (or the
unshare -mwrapper used today) then setsBREENIX_GATE_TMPto a per-clone path instead of relying on mount-namespaceisolation nobody is required to remember. Any other script that reads these
same output directories downstream (verdict/tally readers) needs the same env
var threaded through, or it will look in
/tmp/...while the boot itselfwrote to
$BREENIX_GATE_TMP/....The R18 lesson
A gate's PASS verdict was read off a directory, not off the lane's own
build. The fix above prevents the directory collision, but the harness should
also not trust "this directory contains a passing tally" on faith — the
collision here was only caught because someone diffed the serial's printed
launch strings against the boot image's own embedded strings after the fact.
A verdict reader should tie a PASS to the lane's own build hash (e.g. embed or
check the kernel ELF hash / a per-run token in the serial output it just
produced) so a future collision fails loudly instead of silently scoring the
wrong kernel as green.
Sources
docs/planning/green-program/nic-bus/737-DF-ORACLE-2026-09-04.md(section 2, the collision + how it was caught)docs/planning/green-program/nic-bus/737-FIX-2026-09-04.md(section 8, "mitigated per-run, not fixed")docs/planning/green-program/nic-bus/serials/737-df-fix-2026-09-04/hashes.txt(theunshare -m+ per-clone bind-mount record)