Skip to content

gates(aarch64): the native boot test declares its boot_tests-kernel requirement; CLAUDE.md names the production gate - #815

Merged
ryanbreen merged 2 commits into
mainfrom
gates/aarch64-native-boot-tests-guard
Sep 5, 2026
Merged

gates(aarch64): the native boot test declares its boot_tests-kernel requirement; CLAUDE.md names the production gate#815
ryanbreen merged 2 commits into
mainfrom
gates/aarch64-native-boot-tests-guard

Conversation

@ryanbreen

Copy link
Copy Markdown
Owner

The native ARM64 boot-test gate now refuses a wrong-profile kernel fast (health-811)

Base: main at 9b3dd4af9dc53d2950688f8094a26351703892c.

What was red

docker/qemu/run-aarch64-boot-test-native.sh scores a boot only through
run_single_test's final checks, and one of those checks is unconditional:

if ! grep -qF -x "$INIT_GROUP_REFUSAL_ORACLE_LITERAL" "$OUTPUT_DIR/serial.txt" 2>/dev/null; then
    echo "FAIL: init-group refusal oracle counter marker missing"
    return 1
fi

INIT_GROUP_REFUSAL_ORACLE_LITERAL is the serial line
init_group_refusal_oracle_test() prints
(kernel/src/tracing/providers/teardown.rs:5125), and that function is
#[cfg(feature = "boot_tests")]. The gate also requires the
[INIT_GROUP_WALK:...] marker, emitted by emit_init_group_walk()
(kernel/src/tracing/providers/teardown.rs:1422), which carries the same
#[cfg(feature = "boot_tests")] gate. A kernel built without
--features boot_tests — the shipped production profile — cannot print
either line, because kernel/src/test_framework/mod.rs:60-66 gates the whole
registry/executor module tree, where these functions are registered and
invoked, on that same feature. Since the marker can not be present in such a
kernel's serial output under any boot outcome, a boot of it deterministically
fails run_single_test on "init-group refusal oracle counter marker missing" (or, earlier in the same run, "exec first commit not observed"
if the boot is slow enough that the 24s liveness window elapses first) — this
branch did not run the pre-fix script against a production kernel to count
the failed attempts directly, since the fix lands before the retry loop is
reached, but the deduction above is what the pre-fix script's own
MAX_RETRIES=5 retry loop and its generic
ARM64 BOOT TEST: FAILED (after 5 attempts) banner would have produced.
Nothing in that banner says the kernel was built in the wrong profile; it
reads exactly like a boot regression.

docker/qemu/run-aarch64-boot-test-strict.sh had already been fixed for
this class of failure (its own require_boot_tests_kernel(), added ahead of
this change) but the native script had not.

The fix

(a) require_boot_tests_kernel() guard

Added the same function name and body shape as the strict gate's guard —
same for-loop census of boot_tests-only marker prefixes, same
missing="$missing $marker" accumulation, same
if [ -n "$missing" ]; then ... exit 1; fi structure — inserted right after
the existing check-kernel-no-neon.sh preflight and before the ext2-disk
check, called once at top level as require_boot_tests_kernel "$KERNEL".

The one deliberate difference from the strict gate's copy: this gate's own
verdict convention is the ARM64 BOOT TEST: PASSED / ARM64 BOOT TEST: FAILED banner printed at the bottom of the script on the retry-loop path.
Review R157 correction: the first version of this change routed only
the new missing-marker arm through that banner, leaving this script's two
pre-existing preflight rejections (no kernel found, no ext2 disk found)
exiting bare — the same defect this whole change exists to remove, on a
different arm. Fixed in the same commit as this correction: 3 of 3
preflight arms now print the banner (each with its own parenthetical)
immediately before their exit 1, so every preflight rejection in this
script flows through the verdict shape this script already reports through
on its ordinary retry-exhausted path, not just the marker-census one.

The same principle — route a preflight rejection through the

script's one verdict-reporting mechanism rather than a bare exit — is
what docker/qemu/run-x86-prod-profile-boot-test.sh's report_gate_failure
machinery enforces for that gate, in that mechanism's own idiom: a single
report_gate_failure ...; exit "$exit_code" call site, an ERR trap, and a
reached flag, all checked by
tests/teardown_structure.rs::validate_x86_prod_profile_harness
(docs/planning/green-program/gates/GATE-PREFLIGHT-VERDICT-802-2026-09-05.md).
This native script implements the analogous idea independently, with its
own bespoke banner-before-exit 1 convention, no ERR trap, and no reached
flag, and would NOT itself satisfy that x86-specific structural test — it
rejects any exit statement other than exit "$exit_code", and every
preflight arm in this script exits via a bare exit 1. A downstream
harness grepping for the ARM64 BOOT TEST: FAILED banner text gets a
labeled verdict on all three preflight rejections, not an unlabeled early
exit that reads as a script crash.

The census reuses the strict gate's exact seven marker prefixes
([SCHED_STRAND_ORACLE:, [STRAND_INJECT_ORACLE:, [CENSUS_WIDEN_ORACLE:,
[FUTEX_HANDOFF_ORACLE:, [CTX596_ORACLE:, [TOMBSTONE_JOIN_ORACLE:,
[BOOT_TESTS:) rather than a native-specific pair.
Six of the seven — every ORACLE marker in the list — are general
boot_tests-only profile markers this gate's own run_single_test does not
otherwise score; [BOOT_TESTS: is the exception, since run_single_test
already checks for a [BOOT_TESTS:FAIL line (a check that is a harmless
no-op against a kernel built without the feature, since such a kernel never
emits any [BOOT_TESTS:...] line at all). The six-marker redundancy is
reused here so the guard stays a robust profile detector — a single marker
regressing to a different profile cannot quietly disarm it — instead of a
narrower check that a smaller future refactor could accidentally shrink
toward vacuousness. The gate's actual pinned boot_tests-only markers,
INIT_GROUP_REFUSAL_ORACLE_LITERAL and INIT_GROUP_WALK, are named in the
guard's failure message so the operator sees the concrete reason this gate
specifically needs the feature, not just an abstract marker list.

(b) CLAUDE.md

Two minimal wording changes in the "Test Scripts" / "Standard Workflow"
sections, no other edits:

  • The ARM64: bullet list now says the native and strict scripts each
    need a --features boot_tests kernel, and adds the previously-unlisted
    run-aarch64-prod-profile-boot-test.sh as the production-profile script
    (it builds its own no-features kernel internally, so it needs no build
    line of its own).
  • The cargo build ... kernel-aarch64 command that precedes
    ./docker/qemu/run-aarch64-boot-test-native.sh in "Standard Workflow" now
    includes --features boot_tests, with a one-line comment saying which
    script needs the feature and which one (the prod-profile gate) does not.

Before this change, CLAUDE.md's own documented workflow built a
no-features kernel and then ran the gate that requires boot_tests markers —
the exact mismatch this whole change is about, reproduced by a reader
following CLAUDE.md verbatim.

(c) Anti-vacuity: extend the existing structure-test census

tests/strand_handoff_structure.rs::boot_tests_gates_refuse_a_wrong_profile_kernel
already censused the service-sequence, strict, and full-test gates for this
guard shape (function body non-empty, for marker in '...' ...; do line
with at least MIN_BOOT_TESTS_PROFILE_MARKERS (6) bracketed markers,
grep -aqF inspection, single missing="$missing $marker" accumulation,
single exit 1 in the missing-marker arm, single top-level invocation,
no-NEON preflight ordered first). Added a NATIVE_GATE_PATH constant and
one entry, ("native gate", NATIVE_GATE_PATH), to that test's gate list.

This census

covers the four gate scripts the test already names, not a directory-wide
discovery: those four scripts cannot silently lose this guard shape, but a
new fifth script is invisible to this list until someone adds it.

(d) Review R157: the top-of-script build hint, and one honesty fix

N1 (major). The script's own "No ARM64 kernel found" arm — the very
first preflight check, at the top of the file, before
require_boot_tests_kernel() is even defined — still told the operator to
build a kernel WITHOUT --features boot_tests: the exact kernel the guard
60 lines below then refuses. 3 of 3 peer boot_tests-requiring aarch64
gates (service-sequence, strict, full-test) already carried the feature in
that same arm; this script was the one exception. Fixed by adding
--features boot_tests to that hint, with a one-line comment pointing at
require_boot_tests_kernel() as the reason.

The pre-existing ratchet,
strict_gate_build_hint_enables_boot_tests, only ever checked the strict
gate's build hint, so it could not have caught this. Renamed to
boot_tests_gate_build_hints_enable_boot_tests and generalized to loop over
both the strict and native gates. The first pass at the generalized
assertion (.any(build_hints, contains "--features boot_tests")) was
itself vacuous against exactly this bug: the native gate has a second
cargo build echo line, inside require_boot_tests_kernel()'s own
missing-marker arm, which already carried --features boot_tests before
this round — so "at least one hint has the feature" was already true even
with the top-of-script hint broken. Strengthened to .all(...), which
requires every build-hint line in the gate to carry the feature.
Mutation-proven both ways in this round: reverting only the top-of-script
hint to drop --features boot_tests (leaving the guard's own hint
untouched) reddens boot_tests_gate_build_hints_enable_boot_tests with
"native gate every build hint must enable --features boot_tests"; restoring
it returns the suite to green (38/38 in
tests/strand_handoff_structure.rs, plus tests/exec_lock_order_structure.rs
44/44 and tests/teardown_structure.rs 83/83, all of which also read this
script's text and were otherwise unaffected).

N3 (minor). The in-script comment ahead of require_boot_tests_kernel()
had claimed the ratchet proved the banner echoes print "by construction,"
citing the same exit 1-count check the ratchet actually performs. The
ratchet proves the arm has exactly one exit 1 line; it does not inspect
the echo lines above it, so deleting the three banner echoes alone (proven
by mutation, then reverted) leaves the ratchet green. Reworded the comment
to state only what the ratchet checks, and to say plainly that the banner's
presence is a maintained convention, not a ratcheted one.

Mutation proof the added census line is load-bearing

With the fix applied and require_boot_tests_kernel "$KERNEL" temporarily
deleted from a scratch copy of the native script (not committed — restored
before this branch's tests were re-run), cargo test --test strand_handoff_structure boot_tests_gates_refuse_a_wrong_profile_kernel
failed:

thread 'boot_tests_gates_refuse_a_wrong_profile_kernel' panicked at tests/strand_handoff_structure.rs:1755:9:
assertion `left == right` failed: native gate must invoke the boot-tests profile guard exactly once at top level
  left: 0
 right: 1

Restoring the file and re-running the same test returned test result: ok. 1 passed.

Proofs

Fail-fast on the production (no-features) kernel

Built with cargo build --release --target aarch64-breenix-kernel.json -Z build-std=core,alloc -Z build-std-features=compiler-builtins-mem -p kernel --bin kernel-aarch64 (no --features), verified with
scripts/check-kernel-no-neon.sh (PASS: 0 FP/SIMD load/store instructions), then run once against
docker/qemu/run-aarch64-boot-test-native.sh:

Error: .../target/aarch64-breenix-kernel/release/kernel-aarch64 was not built with --features boot_tests.
  Missing boot_tests-only marker literal(s): [SCHED_STRAND_ORACLE: [STRAND_INJECT_ORACLE: [CENSUS_WIDEN_ORACLE: [FUTEX_HANDOFF_ORACLE: [CTX596_ORACLE: [TOMBSTONE_JOIN_ORACLE: [BOOT_TESTS:
  This gate pins INIT_GROUP_REFUSAL_ORACLE_LITERAL and the INIT_GROUP_WALK
  marker, both boot_tests-only, so every boot below would fail on
  'marker missing' after 5 retries -- not a kernel red.
  Rebuild with:
    cargo build --release --features boot_tests --target aarch64-breenix-kernel.json -Z build-std=core,alloc -Z build-std-features=compiler-builtins-mem -p kernel --bin kernel-aarch64

=========================================
ARM64 BOOT TEST: FAILED (kernel is not a boot_tests build)
=========================================

Exit status 1, wall time 2.08s (time on the run). pgrep -fl qemu-system-aarch64 | wc -l reported 0 immediately before this run and 0
immediately after it, so this run itself launched no QEMU process; this
branch did not run the pre-fix script against this same production kernel to
directly count how many qemu-system-aarch64 processes its 5-attempt retry
loop would have launched (see "What was red" above for the deduction that it
would be 5, one per MAX_RETRIES attempt, each on a 30s timeout).

Pass on the boot_tests kernel

Rebuilt with --features boot_tests added (userspace ELFs built first via
userspace/programs/build.sh --arch aarch64, since the boot_tests registry
embeds simple_exit.elf at compile time), re-verified with the same
no-NEON guard, ext2 image created with scripts/create_ext2_disk.sh --arch aarch64, then run once:

Attempt 1/5...
SUCCESS

=========================================
ARM64 BOOT TEST: PASSED
=========================================

Exit status 0, passed on the first attempt (no retry needed).

Pass on the production-profile gate

docker/qemu/run-aarch64-prod-profile-boot-test.sh, which builds its own
no-features kernel and boots it, run once:

PASS: production profile reached bsshd with the futex oracle seam absent
Observed: [FUTEX_HANDOFF_ORACLE_DRIVER:seam_absent:probe=-110]
Observed: [init] futex_handoff_oracle exited pid=6 code=0
Observed: bsshd: listening on 0.0.0.0:2222
Observed kernel oracle marker count: 0
...
Observed: [TTBR0_ASID_CENSUS:untagged=0:tagged=23431:kernel=26892:cleared=49512]
Observed crash marker count: 0

Exit status 0.

Structure-test suites

Every file in tests/*_structure.rs (30 files at this head) run
individually with cargo test --test <file>: 30 of 30 green, 562 test
cases total (aarch64_testing_profile_structure 2,
block_request_lifetime_structure 12, context_restore_structure 97,
coreproof_component_h_structure 5, coreproof_coverage_structure 4,
coreproof_mutation_register_structure 5, coreproof_sites_structure 4,
degenerate_transfer_fd_validation_structure 4,
dispatch_path_lock_free_structure 4, dispatch_strand_census_structure 7,
dma_and_log_sink_structure 4, entry_point_df_structure 5,
exec_lock_order_structure 44, exit_tally_structure 6,
ext2_lock_structure 36, fork_lock_order_structure 10,
green_program_envelope_structure 14, loopback_pump_structure 72,
masked_binary_load_structure 4, mmap_floor_structure 9,
net_lock_structure 19, poll_tcp_gate_wiring_structure 3,
preempt_bracket_structure 8, serial_line_atomicity_structure 9,
signal_eintr_predicate_structure 2, strand_handoff_structure 38,
syscall_return_register_structure 6, teardown_structure 83,
ttbr0_shadow_reconciliation_structure 32, tty_oracle_structure 14),
0 failed.

Claim-lint

claim-lint: scripts/claim-lint.py -> exit 0

plus a --files/--commit-msg run recorded in the round notes for this
branch's own diff and commit message.

What is NOT claimed

  • This is a gate/tooling fix, not a kernel change. kernel/ was not
    touched; no kernel behavior changed.
  • The seven marker prefixes the guard's census reuses are general
    boot_tests-profile detectors, not a claim that run-aarch64-boot-test-native.sh
    itself scores SCHED_STRAND_ORACLE, FUTEX_HANDOFF_ORACLE,
    CENSUS_WIDEN_ORACLE, CTX596_ORACLE, or TOMBSTONE_JOIN_ORACLE — it
    does not. The two markers this gate does score and require are
    INIT_GROUP_REFUSAL_ORACLE_LITERAL and INIT_GROUP_WALK, named
    separately in the guard's own failure message.
  • The three boot proofs above are each a single observation (one fail-fast
    run, one pass on boot_tests, one pass on the production-profile gate),
    not a multi-boot soak. The native gate's own MAX_RETRIES=5 retry
    mechanism and the strict gate's iteration count remain the tools for
    measuring flake rate; this change does not add or remove either.
  • This change does not touch docker/qemu/run-aarch64-boot-test-strict.sh,
    run-aarch64-service-sequence-gate.sh, run-aarch64-full-test.sh, or any
    x86 gate script. Their preflight/verdict shapes are unchanged.
  • x86 boot gates and the run-x86-prod-profile-boot-test.sh verdict-
    discipline ratchet (x86_production_profile_gate_verdict_discipline_holds)
    were read for the pattern this fix follows but were neither run nor
    modified as part of this change.
  • CLAUDE.md changes are limited to the two lines described in (b) above; no
    other section of CLAUDE.md was reviewed or edited as part of this task.

…ile kernel fast

docker/qemu/run-aarch64-boot-test-native.sh unconditionally required
INIT_GROUP_REFUSAL_ORACLE_LITERAL and the INIT_GROUP_WALK marker, both
emitted only by #[cfg(feature = "boot_tests")] code
(kernel/src/tracing/providers/teardown.rs, gated at the module level by
kernel/src/test_framework/mod.rs:60-66). A production kernel built without
that feature can not print either marker, so each of its 5 MAX_RETRIES
boots ran into a generic "marker missing" failure with nothing saying the
kernel's build profile was the cause (deduced from the cfg gate, not
separately measured against the pre-fix script -- see the doc's "What was
red" section).
claim-lint:ok: structural claim about #[cfg(feature = "boot_tests")]
resolved by direct citation at kernel/src/tracing/providers/teardown.rs:5125-5126
and :1422-1423, and kernel/src/test_framework/mod.rs:60-66; a build without
that feature does not compile the cited code, so it cannot emit the marker.

Added require_boot_tests_kernel(), the same shape as the strict gate's
existing guard (a census of 7 boot_tests-only marker prefixes, one
missing="$missing $marker" accumulation, one exit 1 in the missing-marker
arm), called once before the retry loop. That arm prints the script's own
ARM64 BOOT TEST: FAILED banner ahead of its exit, so a downstream harness
watching for that banner still sees a labeled verdict on this failure mode
instead of an unformatted early exit.

CLAUDE.md's ARM64 test-script list and its Standard Workflow build command
now say the native/strict gates need a --features boot_tests kernel and add
the previously-unlisted production-profile gate, which builds its own
no-features kernel and needs no change here.

tests/strand_handoff_structure.rs::boot_tests_gates_refuse_a_wrong_profile_kernel
now censuses the native gate too (a NATIVE_GATE_PATH entry alongside the
existing service-sequence/strict/full-test rows), so a future aarch64 gate
script shipping without this guard shape reddens that test. Confirmed by
deleting the guard invocation in a scratch copy of the script (test failed:
"native gate must invoke the boot-tests profile guard exactly once at top
level", left: 0, right: 1) and restoring it (test passed).

Proofs: a no-features production kernel run against the native gate now
fails in 2.08s with the "was not built with --features boot_tests" verdict
and zero QEMU processes launched, instead of 5 retries; a --features
boot_tests kernel passes the native gate on attempt 1/5; the
production-profile gate (run-aarch64-prod-profile-boot-test.sh) passes; all
30 tests/*_structure.rs suites are green, 562 cases, 0 failed. Detail,
including what is not claimed, is in
docs/planning/green-program/gates/NATIVE-GATE-GUARD-2026-09-05.md.

claim-lint: scripts/claim-lint.py -> exit 0
claim-lint: scripts/claim-lint.py --files docs/planning/green-program/gates/NATIVE-GATE-GUARD-2026-09-05.md -> exit 0

Co-Authored-By: Ryan Breen
Co-Authored-By: Claude Code
…claim, and three minors

6 of 6 review findings on 329ab9c (the native-gate wrong-profile-kernel
guard) closed in this round.

N1 (major): the script's own "No ARM64 kernel found" arm -- the first
preflight check, above require_boot_tests_kernel() -- still told the
operator to build a kernel WITHOUT --features boot_tests, the exact
kernel the guard 60 lines below then refuses. 3 of 3 peer aarch64 gates
already carried the feature in that arm; this script was the exception.
Added --features boot_tests to the hint. The pre-existing ratchet
(strict_gate_build_hint_enables_boot_tests) only ever checked the
strict gate, so it could not have caught this; generalized it to
boot_tests_gate_build_hints_enable_boot_tests, looping over both
gates. Its first draft (.any(hint contains the feature)) was itself
vacuous against this exact bug -- the native gate's second build-hint
line, inside require_boot_tests_kernel()'s own missing-marker arm,
already carried the feature, so "at least one hint has it" was already
true with the top-of-script hint broken. Strengthened to .all(...).
Mutation-confirmed both ways: reverting only the top-of-script hint
reddens the test ("native gate every build hint must enable
--features boot_tests"); restoring it returns the suite to green.

N2 (major): NATIVE-GATE-GUARD-2026-09-05.md and 329ab9c's own commit
body claimed "a future aarch64 gate script cannot ship without this
guard passing the same census" / "reddens that test". False --
boot_tests_gates_refuse_a_wrong_profile_kernel iterates a literal
four-path list, not a directory scan; a new fifth gate script is
invisible to it. docker/qemu/run-aarch64-arma609-arm.sh already
demonstrates the gap (its own require_boot_tests_kernel(), 5-marker
census, 0 references in tests/*.rs). Narrowed the doc's claim to
what the census actually covers; the commit body is left as pushed
per house style (pushed history stays intact) -- this commit's own
body states the correction instead.

N3 (minor): the script comment ahead of require_boot_tests_kernel()
claimed the ratchet proves the banner echoes print "by construction."
The ratchet only proves the arm's exit-1 count; deleting the three
banner echoes alone (mutation-confirmed, then reverted) leaves it
green. Reworded to state only what the ratchet checks.

N4 (minor): only the new missing-marker arm routed through the
PASSED/FAILED banner; the script's other two preflight rejections (no
kernel, no ext2 disk) still exited bare. Routed both through the same
banner convention, live-proven: ran the gate with no kernel present
(prints the new build hint + "FAILED (no ARM64 kernel found)"), then
with a --features boot_tests kernel built but no ext2 image (prints
"FAILED (ext2 disk not found)"), then end to end with both present
(PASSED, attempt 1/5, zero orphaned qemu-system-aarch64 processes
before or after).

N5 (minor): docs/planning/ARM64_TEST_CATALOG.md's "Running Tests"
recipe still built --features testing before running this gate, which
now hard-fails on the guard (testing and boot_tests are independent
Cargo features). Corrected to --features boot_tests, with a note that
the full-test-suite line builds its own --features testing kernel
internally and is unaffected.

N6 (nit): the doc's "is what run-x86-prod-profile-boot-test.sh's
report_gate_failure machinery enforces" sentence could read as this
script satisfying that ratchet. It would not --
validate_x86_prod_profile_harness rejects any exit other than
exit "$exit_code", and this script's 3 of 3 preflight arms exit via
bare exit 1. Reworded to state the shared principle and disclose the gap
explicitly.

Verification: cargo test across strand_handoff_structure (38/38),
exec_lock_order_structure (44/44), and teardown_structure (83/83);
the full 30-binary tests/*_structure.rs family, 562/562, 0 failed
(same total the prior commit's own proof cited). Live boots on this
Mac (aarch64 native): missing-kernel arm, missing-ext2 arm, and a full
--features boot_tests pass (attempt 1/5, PASSED), each confirmed by
running the script directly; no qemu-system-aarch64 process was ever
left running across any of them.

claim-lint: scripts/claim-lint.py -> exit 0

Co-Authored-By: Ryan Breen
Co-Authored-By: Claude Code
@ryanbreen
ryanbreen merged commit 8a90e3b into main Sep 5, 2026
@ryanbreen
ryanbreen deleted the gates/aarch64-native-boot-tests-guard branch September 5, 2026 11:28
ryanbreen added a commit that referenced this pull request Sep 5, 2026
Pulls in PR #815 (aarch64 native-boot-test gate guard, review r157) from
main; no file overlap with this branch's 7 gate scripts, doc, serials, or
tests/teardown_structure.rs -- main's own changes this round touched
docker/qemu/run-aarch64-boot-test-native.sh and tests/strand_handoff_structure.rs
instead, so this merge carries no conflicts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant