Skip to content

x86: RECLAIM_DRAIN_ACTIVE strands true and production deferred reclamation stops for the rest of the boot (18 reclaims leaked, main and branch alike) #653

Description

@ryanbreen

Symptom

On x86_64 in the boot_tests profile, production deferred reclamation stops for the rest of the boot. RECLAIM_DRAIN_ACTIVE is left true, so every subsequent reclaim_deferred_process_resources() takes the nested-drain refusal arm and returns without doing any work. The idle loop calls that function once per iteration, so RECLAIM_DRAIN_NESTED_REFUSED climbs at ~1000/second for as long as the machine idles, and the pending queue never empties.

Measured with an instrumented probe (temporary, not committed) that samples from idle_loop() every 2 s after the userspace phase ends, holding QEMU 90 s past the last userspace exit:

[PROBE:t=43254:live_q=18:parked_q=0:owner=0:drain_active=true:ctx_viol=0:nested=410]
[PROBE:t=45254:live_q=18:parked_q=0:owner=0:drain_active=true:ctx_viol=0:nested=2409]
...
[PROBE:t=61254:live_q=18:parked_q=0:owner=0:drain_active=true:ctx_viol=0:nested=18409]
  • live_q=18 — eighteen deferred process reclaims are pending and never processed. Their page tables and frames are never released.
  • parked_q=0 — nothing is parked, so the park/unpark machinery is not involved and the age backstop never applies.
  • owner=0 — no BootReclaimTestGuard leak; the boot-test owner is clear.
  • ctx_viol=0, pm_held=false, sched_scope=false — the drain is not being refused for context reasons.
  • drain_active=true, nested climbing by exactly 2000 per 2 s — the refusal is the RECLAIM_DRAIN_ACTIVE compare-exchange in reclaim_deferred_process_resources() (kernel/src/task/process_task.rs).

Pre-existing, by A/B

The same probe applied to main @ ca147f94 reads live_q=18:parked_q=0:owner=0:drain_active=true:ctx_viol=0:nested=18420 at t=74s — identical shape and identical queue depth. This is not introduced by any in-flight branch.

Why it matters

Eighteen exited processes per boot keep their deferred resources on x86. It also means no production retirement receipt ever completes during the x86 boot-test userspace phase, which is why every live waitpid reap on that arch wins the race against retirement.

It surfaced through P6a PR-2's tombstone census: with the two-event join installed, four of those rows are reaped-but-never-retired and stay resident, and [TOMBSTONE_CENSUS:resident=4:removed=2:...] names them. The strand itself is pre-existing, as above, but the row retention is P6a's, not merely a symptom made visible: on main, complete_wait removes the Process row unconditionally at the reap, so those four rows were freed even while the underlying reclaim was stranded. P6a's join instead fail-closes row removal on a retirement that x86 never completes, so the rows are retained forever rather than freed-and-invisible. Two consequences: this issue is now a prerequisite for x86 row-removal correctness under P6a's join, not only for x86 evidence of return-to-zero; and the probe above only ever ran in the boot_tests profile — there is no x86 production-profile teardown gate (the #540 gap) — so whether the same per-process row retention occurs in production x86 is unmeasured and unknown.

Where to look

reclaim_deferred_process_resources() sets RECLAIM_DRAIN_ACTIVE with a compare-exchange, runs one pass, and stores false afterwards. Any path that leaves the pass without reaching that store — a context switch taken from inside the pass, or a fault that IRETs into idle_loop — strands the flag permanently, because nothing else ever clears it. The sibling BOOT_RECLAIM_TEST_OWNER documents exactly this failure mode as deliberate ("an abandoned drain leaves the flag set on purpose ... a bounded leak instead of a hard hang"), but for the production flag the consequence is that reclamation is off for the remainder of the boot with no counter naming the transition — only the nested-refusal counter climbing.

Worth considering: an owner identity plus a liveness/epoch check rather than a bare boolean, so an abandoned drain is recoverable and attributable rather than terminal.

Repro

docker/qemu/run-x86-boot-tests.sh 1 on the beast x86 container, with the QEMU kill deferred ~90 s past TEST RUNNER: in serial_kernel.txt, and a probe printing RECLAIM_DRAIN_ACTIVE / queue depths from idle_loop().

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions