Process teardown quiescence: free address space and kernel stack only after the CPU provably leaves them

[ryanbreen]

Summary

Fixes an aarch64 SMP teardown race where a terminated thread's kernel stack
and address-space root could be reclaimed and reused while a CPU was still
provably executing on them. The reused 64 KiB stack slot got zero-scrubbed
under a live exception frame, and a still-live TTBR0 root got freed out from
under a peer CPU mid-ERET. Both produce an impossible return frame: ELR=0x1
(garbage return address), spsr=0x134598 (nonsense SPSR), captured live in
logs/parallels-launcher-test/run-20260802-045109/serial-excerpt.txt:1097
during a bterm fork/exec teardown — the exact crash class this PR closes.

Reconciled mechanism: teardown now frees the kernel stack slot and the
process's page-table root only after the CPU has provably left them, not
merely "probably":

• Kernel stack (in-context teardown): a retiring stack slot is excluded
  from reuse while any online CPU's kernel_stack_top/user_rsp_scratch
  still names it, and requires two full scheduling-epoch bumps
  (saturating_add(2), not 1) past termination. One bump can be recorded
  by the very handoff still tearing the thread down (the epoch counter is
  bumped at the top of check_need_resched_and_switch_arm64, before
  dispatch_thread_locked repoints the stack pointers and before the
  boot.S epilogue finishes reading the old stack) — two bumps guarantees a
  second, later exception entry, which cannot happen while the first
  handoff's ERET/register-restore is still in flight.
• TTBR0 root (per-CPU shadow re-install): before a deferred SIGSEGV exit
  or a normal sys_exit publishes teardown, the faulting/exiting CPU calls
  quiesce_ttbr0_for_exit(), installing the kernel TTBR0 locally and
  clearing both saved_process_ttbr0/next_ttbr0 shadows. Reclamation then
  snapshots those shadows across every online CPU (is_ttbr0_root_live)
  and only frees the root once no peer CPU retains it and the two-epoch
  stack grace above has elapsed. The same-CPU path stays immediate when local
  quiescence already leaves no peer holding the root.
• sys_exit's terminal WFI was replaced with the existing inline scheduler
  pivot, so Terminated is published only after assembly has moved SP off
  the dying stack onto the per-CPU scheduler stack and a successor has been
  dispatched.

STEP 0 / 1 / 2

• Step 0 (120eb569, diag): postmortem-evidence ordering — moves
  high-value teardown evidence ahead of trace-buffer dumping, claims every
  postmortem section independently (a nested abort skips only the section
  already in progress), derives reusable-stack classification from allocator
  constants, stamps/reports the owning tid per stack slot, and selects
  deferred SIGSEGV cleanup from the faulting frame's stamped owner instead of
  the scheduler's current idle identity. Record-only; does not change
  teardown ordering.
• Step 1 (0dc2f440, fix): makes the impossible return states
  unrepresentable on the covered paths — user context restore forces EL0t
  mode; the inline-schedule path stores the kernel resume mask in
  Thread.inline_schedule_spsr (leaving context.spsr_el1 paired with the
  saved user ELR); both Rust idle redirects select kernel TTBR0 via
  next_cr3 and clear saved_process_cr3; the sync/IRQ/syscall epilogues
  reject EL1 returns below KERNEL_VIRT_BASE and record the rejected
  ELR/SPSR in fixed per-CPU slots using branch-only stores (no UART on the
  hot path).
• Step 2 (b34448a9 + 6f66f37f, fix): the actual quiescence-before-
  reclamation machinery described above (kernel-TTBR0 install before exit/
  exec/fault teardown, two-epoch stack grace, cross-CPU TTBR0-shadow scan
  before root reclamation, deferred-reclaim list drained from sys_fork).
  6f66f37f is a codex fix-forward round closing two review-blocking holes
  found in b34448a9: (1) the epoch race described above (saturating_add(1)
  → saturating_add(2)), and (2) drain_deferred_fault_sigsegv_exits()
  draining any CPU's buffer and tearing down the page table with no
  cross-CPU quiescence — closed by the quiesce_ttbr0_for_exit +
  is_ttbr0_root_live design above.

Gold-master adjacency signoff

Step 1 widens the pre-ERET return-privilege guard conditions, which sit
immediately adjacent to the gold-master-frozen ERET/dispatch regions called
out in docs/planning/cpu0-user-guard-autopsy/README.md. Per that autopsy's
requirement, the design was read before editing, and this change does
not touch aarch64_enter_exception_frame, ISB placement, ERET
instruction placement, or ERET ordering — only the existing pre-ERET branch
conditions and branch-only per-CPU record stores. Independently re-verified
at final HEAD 6f66f37f via git diff against 2b98725d: idle_loop_arm64,
the EL0-dispatch banner, aarch64_enter_exception_frame (incl. ISB/ERET
ordering), gic.rs's SGI-enable block, timer_interrupt.rs, and every
Tier-1 x86_64 file are byte-identical / untouched. This still needs
project-owner signoff on the guard widening before/at merge, per the
autopsy's own requirement — flagging explicitly rather than asserting it
away.

Gates (honest numbers)

• Builds: x86_64 release and aarch64 release both re-verified zero-warning
  at exact HEAD 6f66f37f (aarch64 also links clean, no CONDBR19 relocation
  failures) immediately before opening this PR.
• Native SMP=4 aarch64 QEMU harness: inconclusive, not 10/10 — the harness
  reports "Userspace not detected" on this branch, but that reproduces
  identically on baseline 2b98725d after an ext2 rebuild (tracked
  pre-existing gap: bd breenix-5up, /bin/bwm spawn EIO stalls the
  userspace-completion marker even on a healthy kernel boot). No
  UNHANDLED_EC/DATA_ABORT/INSTRUCTION_ABORT/FATAL_POSTMORTEM/panic markers on
  any attempt. One ad-hoc QEMU smoke boot at 6f66f37f did complete fully:
  4 CPUs online, init/bsshd/heartbeat/xhci_counters spawn, xhci_counters
  exits(0) cleanly exercising the modified exit path, 20+s with no fault.
• Parallels boot gate: 3/3 clean (uptimes 113–120s, CPU0 ticks
  50,000–65,000+, 105–120 heartbeats each, zero fault markers, VMs
  force-stopped and verified stopped).
• Parallels launcher streak: 14/14 attempts run sequentially
  (--no-build, fresh VM each, screen unlocked + VM-stopped verified before
  every attempt). 13/14 RESULT: PASS clean (inject_retries=0); 1/14
  (attempt 7) ENV: HOST_INPUT_WEDGE — a documented host-side Parallels
  USB-HID dispatcher wedge (probe key never incremented in 10s), not a kernel
  fault and not screen-lock; Breenix was not exercised that attempt per the
  harness's own classification. Zero kernel-fault markers across all 14
  attempts. Streak sequence: attempts 1–6 clean (streak 6), attempt 7 the
  environmental break, attempts 8–14 clean (streak 7). cleanStreakMax=7;
  the 10-consecutive target was not reached — mathematically foreclosed
  once attempt 7 broke the streak with only 7 attempts left. Per the literal
  rubric (which names screen-lock specifically as the allowed environmental
  exception, and this was a different cause) this reports RED, while
  noting the underlying kernel fix showed 0/14 kernel faults — the miss is
  test-harness/host environmental noise, not evidence of a regression.
• Prior context (the bug this PR fixes): on main 2b98725d, a 90-minute
  soak was clean (no UNHANDLED_EC/DATA_ABORT/panic/POSTMORTEM/CPU0 ALARM),
  confirming baseline health. In the same session, a follow-on
  final-acceptance launcher streak then hit a kernel fault on attempt 1/14
  — cascading INSTRUCTION_ABORT/EL1_INLINE_ABORT/FATAL_POSTMORTEM during
  bterm fork/exec, inject_retries=0 (not a harness artifact), producing
  the ELR=0x1/spsr=0x134598 impossible frame referenced above. That
  fault is what motivated this branch.

Remaining follow-ups (not fixed here, tracked separately)

• bd breenix-5up — pre-existing /bin/bwm spawn EIO on native SMP=4
  aarch64 boot prevents the userspace-completion marker even on an otherwise
  healthy boot; reproduces on baseline 2b98725d too, unrelated to this fix.
• CPU0 timer-rate-dip curiosity: during the prior r9 90-minute soak on main,
  a transient ~4-minute window (~04:43–04:47 EDT) saw the CPU0 timer-tick
  rate drop from a steady ~500 Hz to ~78–90 Hz before recovering to ~624 Hz.
  No alarm fired and ticks never stalled, but flagging given this project's
  CPU0-timer fragility history. Not reproduced or investigated further here.
• Explicitly out of scope per the commits: Box<ProcessPageTable> has no
  Drop impl and does not deallocate its L0/L4 (or lower) table frames on
  the active-process path (pre-existing leak, tracked as breenix-mt0);
  refcounted lifetime for non-running CLONE_VM owners is the separate
  Step 3 (breenix-m25).
• Non-blocking nits carried over from review (not fixed in this branch):
  dump_all_eret_frame_anomaly_snapshots prints the raw slot-encoded
  LAST_DISPATCHED_TID as owner_tid_canary (decodes to a garbage tid in
  postmortems); Thread.inline_schedule_spsr is currently write-only (no
  reader yet); SCHEDULING_EPOCHS is 8×AtomicU64 sharing one 64B cache
  line, RMW'd by every CPU on every exception return (false-sharing on the
  hottest path); the EL0 arm of the old elr<0x1000 guard was dropped; the
  (spsr&0xF)!=0 clause of the [BUG] dispatch_thread guard is now dead
  code.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com