fix(fs,#728): park instead of spin on contended ext2 lock -- aarch64 proven, x86 unproven by capture

[ryanbreen]

Refs #728

This PR does NOT close #728. It lands the park-instead-of-spin fix for the
ext2 lock-discipline livelock, proven on aarch64, but the x86 half of
#728 — the arch the bug was originally reported on — remains unproven
by direct oracle capture. #728 stays open after this merges; see
"Honest scope" below for exactly what is and isn't established.

What this PR does

Replaces ext2's contended ROOT_EXT2/HOME_EXT2 lock acquisition —
previously an unconditional, non-yielding busy-spin with preemption
effectively disabled — with a park-on-contention path
(ext2_acquire/ext2_acquire_write in kernel/src/fs/ext2/mod.rs),
falling back to the original spin whenever parking is unsafe
(ext2_lock_can_sleep() false: no current thread, interrupt context,
IRQs masked, more than one nested preempt-disable, or — aarch64 only —
the timer not yet initialized). This closes the livelock shape #728
describes: a contended acquisition busy-spinning denies the CPU the
actual holder's own completion needs, so the holder can never release.

Honest scope — aarch64 proven, x86 unproven by capture

aarch64: proven two-sided, with a positive control.

• RED (pre-fix code + the same harness): EXT2_LOCK_SPIN_STALL lock=ROOT_EXT2_write fires 3 of 3 times, gate FAILs.
• GREEN (fixed code): [LOCKRACE:COMPLETE:pass=2:fail=0], both
  filesystems PASS, EXT2_LOCK_PARKS delta observed nonzero on all 4
  of 4 reruns performed across round 2 and this closure round (130,
  134, and 126 total parks respectively — 126 at the merged bytes used
  to land this PR) — proof the park path was actually entered, not
  merely that no stall happened to occur — kernel live on a line
  genuinely after COMPLETE. Reproduced fresh at merged bytes as part
  of this closure round.
• Service-sequence battery: 49/50 GREEN at merged fix bytes (one
  pre-existing, attributed, unrelated flake), plus a dedicated 25/25
  clean rerun isolated from concurrent local contamination.

x86: park path is code-traced correct; its runtime effect is
UNCAPTURED after four independent oracle attempts across three
rounds.

• The fix is traced end-to-end against real x86 syscall-entry state:
  ext2_lock_can_sleep()'s x86 arm requires
  current_thread_id().is_some() && !in_interrupt() && preempt_count() == 1, which is exactly the state at all ~44 x86 fs/syscall
  acquisition sites per the round-2 review's own independent trace of
  entry state (syscall/entry.asm's cli-then-no-sti means IF=0
  throughout, matching the two established precedents this fix mirrors
  — block_request_gate_can_sleep() and
  syscall_sleep_path_available(), neither of which checks IF on x86
  either). A parked x86 thread's only path back onto the CPU is
  can_schedule()'s (preempt_count == 0 && need_resched) clause
  (BlockedOnIO is deliberately excluded from that function's other
  admission clauses, per x86: self-documented gap between preempt_count and can_schedule() during boot-thread disk-completion busy-spin #666/x86 boot thread starves: timer preempts it during busy-spin disk-completion wait, so test_exec self-tests may not complete #508); schedule_current_wait()'s single
  preempt_enable() is what reaches that 0 from the predicate's exact
  == 1, which is why the ratchet now pins == 1 rather than > 0
  (see kernel/src/fs/ext2/mod.rs's doc comment on
  ext2_lock_can_sleep).
• The direct oracle (docker/qemu/run-ext2-lock-race-gate.sh --x86)
  has never produced an EXT2_LOCK_PARKS > 0 observation, a
  [LOCKRACE:COMPLETE:...] line, or any other terminal signal on
  either GREEN or RED, in four attempts (round 1's prove slot, round
  2's fix + prove slots, and a dedicated host-load-corrected recapture
  round). All four end the same way: the leg is reached, holder/
  contender kthreads are actively scheduling, and the boot is killed
  unfinished after tens of minutes.
• This is now understood to be an oracle-pace defect, not evidence
  either way about the fix. The recapture round measured the leg's
  post-spawn kernel-log line-advance rate directly: ~12–13.5s/line,
  flat across a real host-load swing (1-minute load average 2.28 →
  24.21 → back down). A rate that doesn't move when host load moves
  by an order of magnitude rules out the host-contention explanation
  the prior two rounds carried, without establishing what the real
  cause is. Filed as x86: ext2-lock-race oracle pace is pathological (~12-13.5s/line) and load-independent, blocking #728 x86 capture #748, with the full measurement, evidence
  paths, and diagnostic next steps — x86 oracle capture for Both arches: concurrent ext2 read-park vs write-spin is a livelock shape (root & home filesystems) #728 is
  blocked on x86: ext2-lock-race oracle pace is pathological (~12-13.5s/line) and load-independent, blocking #728 x86 capture #748 being diagnosed first.
• What does bear on x86, indirectly: run-boot-parallel.sh -smp 1
  (the historical repro's own real-sys_mkdir-through-syscall-entry
  shape) ran clean 10 of 10 boots at landed fix bytes across this
  branch's rounds — zero occurrences of the Both arches: concurrent ext2 read-park vs write-spin is a livelock shape (root & home filesystems) #728 stall shape. This is
  an absence-of-stall result, not proof the park path executed (no
  park counter is available via this harness); it is corroborating,
  not conclusive.

Net: the x86 code fix is traced correct by static analysis and
mirrors this kernel's own established precedents; its dynamic,
under-load behavior on x86 has not been observed. Do not read this PR
as proving #728's x86 half — that is exactly the review-B2 mistake an
earlier round of this branch made and this round corrected (see
"Corrections carried in this PR" below).

What else this PR carries (from the review rounds)

• Deleted a fabricated evidence record. An earlier round's gate
  header claimed a x86 pass=1:fail=0 GREEN that no archived serial
  or gate-stdout capture from any round supports (review finding B3),
  with an internally-inconsistent description alongside it. Deleted;
  replaced with what was actually observed (see the gate script header
  and x86: ext2-lock-race oracle pace is pathological (~12-13.5s/line) and load-independent, blocking #728 x86 capture #748).
• Two-sided oracle repairs (review finding B4). The race harness
  (kernel/src/fs/ext2_lock_race.rs) previously could report a false
  PASS on a setup failure and had no way to distinguish "no stall
  happened" from "parked and never woke up, still inside its own
  bounded timeout." Every code path now prints a matching
  :race:verdict= line and is scored on an explicit outcome; a
  positive EXT2_LOCK_PARKS delta is required for PASS, checked both
  in-kernel and independently by the gate script; the "kernel live
  after" liveness check is position-aware (requires the marker
  strictly after the leg's own COMPLETE line in the same
  chronological log).
• Structural ratchet, mutation-proven (review finding B5), 26
  tests (tests/ext2_lock_structure.rs): pins the can-sleep gate +
  spin fallback on both acquisition paths (individually, not just
  combined — closing a blind spot where deleting one function's entire
  park loop still passed), the aarch64/x86 arch split of
  ext2_lock_can_sleep() including the exact == 1 preempt-count
  check, guard-drop release-before-wake ordering, checked-vs-bare
  prepare_to_wait, the mount-check accessors routing through the
  park-capable path instead of a raw .read() (both scoped to those
  three functions and file-wide), and a file-wide census that exactly
  two raw _EXT2.write() sites exist (the C5 mount-time initializers).
  Every property has a hand-mutated negative proving the validator
  actually reddens against the real source, not merely "returns Ok on
  main."
• Wake-path correctness and honesty
  (Ext2ReadGuard/Ext2WriteGuard::drop). A has_waiters() "fast
  path" added in an earlier round was found, this round, to take the
  same lock wake_up_one()/wake_up() already take — saving nothing
  on the uncontended drop and costing a second lock acquisition on the
  contended one. Removed; the comment now states the real
  wake-targeting cost instead (a popped reader can defer an
  upgrade-waiting writer's wake to its own bounded per-round timeout,
  up to 6.4s worst case — see below).
• Corrections carried in this PR (closing out round-2 review
  findings that don't depend on further x86 capture): the gate
  script's RED anti-vacuity claims are corrected to what the archived
  serials actually show (both arches' stalls are ROOT_EXT2_write
  only — the leg wedges on ROOT before HOME ever runs, so "BOTH
  filesystems" was never accurate; neither archive contains a
  soft-lockup detector line); an unsound "RED not reddening under
  contention is itself evidence of a host-driven cause" inference is
  deleted from the round-2 evidence README (the control it was based
  on could not have distinguished its two settings, and x86: ext2-lock-race oracle pace is pathological (~12-13.5s/line) and load-independent, blocking #728 x86 capture #748's
  flat-pace finding independently weakens it); a check_live_after_complete()
  arithmetic guard was hardened against an unexplained
  unbound variable failure hit once in an earlier round's own
  archived RED run; the in-kernel EXT2_LOCK_PARKS doc comment and the
  gate header now state plainly that the counter is a global delta
  across the race window (strong corroboration on this leg's dedicated
  profile, not a per-thread proof).

Known residuals, disclosed rather than fixed here

• A writer now parks while holding the UPGRADED bit across every
  retry round (by design, preserving writer-fairness), so a
  contended write can block readers for up to EXT2_LOCK_PARK_ROUNDS × EXT2_LOCK_PARK_TIMEOUT_NS = 6.4s worst case, where pre-fix it held
  UPGRADED only while actually running. Real, new, user-visible
  latency; not exercised by the service-sequence battery (not a
  contention soak); the alternative (drop UPGRADED between rounds)
  regresses writer starvation, which was ruled non-negotiable for this
  fix.
• Completion::wait_timeout()'s documented "no locks held"
  precondition is still violated at every one of the ~64
  accessor-reaching call sites this fix routes park logic through.
  This fix repairs the acute livelock; it does not remove the
  precondition violation itself. Deferred follow-on (Option A in the
  original Both arches: concurrent ext2 read-park vs write-spin is a livelock shape (root & home filesystems) #728 analysis): drop the guard before the block-device
  wait, re-validate Ext2Fs's in-memory allocator state on reacquire.
  Real filesystem-correctness work, not a locking-primitive change.
• New exposure this fix creates, filed as ext2 lock: SIGKILL to a thread parked in the park path leaks the guard, wedging ROOT_EXT2/HOME_EXT2 forever #746 (open): SIGKILL
  to a thread parked mid-acquisition abandons its stack without
  running the held guard's Drop — terminate_process_threads()
  unconditionally marks every victim thread terminated without
  unwinding it, so a killed thread that was BlockedOnIO inside
  ext2_acquire/ext2_acquire_write wedges ROOT_EXT2/HOME_EXT2 for
  the rest of the boot. Pre-fix, ext2 acquisition never parked, so this
  specific wedge could not happen; the mechanism itself is general (the
  same abandon-on-SIGKILL gap applies to completion.rs, futex.rs,
  and BlockRequestGate) and is scoped as a scheduler/signal-delivery
  fix, not an ext2 change. Confirmed mechanism, file:line citations,
  and a repro sketch are on ext2 lock: SIGKILL to a thread parked in the park path leaks the guard, wedging ROOT_EXT2/HOME_EXT2 forever #746.
• EXT2_LOCK_PARKS is a single global counter, read as a delta
  across a race window rather than attributed per-thread. On the race
  leg's dedicated profile nothing else touches ext2 concurrently, so a
  nonzero delta is strong corroboration in practice, but it is
  corroboration, not a per-thread proof. Making it per-race (recording
  contenders' own tids) is a real improvement, not done here.

Verification at merged bytes

• cargo build --release --features testing,external_test_bins --bin qemu-uefi — 0 warnings/errors.
• cargo build --release --target aarch64-breenix-kernel.json -Z build-std=core,alloc -Z build-std-features=compiler-builtins-mem -p kernel --bin kernel-aarch64 (plain and with
  --features boot_tests,ext2_lock_race) — 0 warnings/errors besides
  the one documented, pre-existing, not-from-this-repo core
  future-incompat notice.
• scripts/check-kernel-no-neon.sh — PASS, 0 FP/SIMD instructions.
• All 22 tests/*structure*.rs suites enumerated fresh from disk
  (including this PR's own ext2_lock_structure, 26/26) — 0 failures.
• docker/qemu/run-ext2-lock-race-gate.sh --aarch64 — PASSED,
  pass=2:fail=0, parks=126 total, kernel live after COMPLETE.
• Branch merged forward with origin/main (picking up x86: exec() syscall (and its production ext2 reader) is ENOSYS in the zero-feature production build #721 and fix(#742,#743): single-source the mmap floor, close MAP_FIXED gap, repair teardown census #744)
  before this verification pass; merge was clean, no conflicts.

Related

• Oracle-pace defect blocking x86 capture: x86: ext2-lock-race oracle pace is pathological (~12-13.5s/line) and load-independent, blocking #728 x86 capture #748
• New SIGKILL/parked-guard exposure this fix creates: ext2 lock: SIGKILL to a thread parked in the park path leaks the guard, wedging ROOT_EXT2/HOME_EXT2 forever #746