backlog: file #1210, the connscale FD peak reports an adopted subtree as the engine

[wshallwshall]

Diagnosis only — no code change. test_connscale_smoke_end_to_end red on #298 with:

fd_count_monotonic: fixed_per_conn@N=24: 344 < prior 3.56e+04 * 0.75

The suspect number is the prior, not the failure. Calibrated against this repo's own published at-scale run (docs/benchmarks/results/2026-07-03-adr0066-pooled-atscale/pooled_ab_atscale.txt):

N=500 -> fd_count_peak 2333     N=1000 -> 3835     N=1500 -> 5335
linear fit ~3.0 handles/connection over an ~833 base  =>  N=12 predicts ~870

observed N=24 -> 344      in band
observed N=12 -> 35,600   6.7x the reading for a FIFTEEN HUNDRED connection engine

No leak produces 35,600 at N=12 and 344 at N=24 minutes later in the same sweep, from two separately-spawned engines. The engine was almost certainly healthy in both arms.

Mechanism: stale-ParentProcessId adoption, not a leak

_walk_descendants BFSes the ppid→children map from the engine root and validates nothing — no creation time, no image name, no cardinality bound. Windows does not clear ParentProcessId when a parent exits, and it recycles PIDs, so any live process whose ParentProcessId equals the engine's recycled PID is adopted along with its entire subtree.

Measured on the maintainer's box (406 processes, 204,882 handles total): seven dead PIDs still have live processes pointing at them, and walking one as if it were a freshly-recycled engine PID summed 144,688 handles. The observed range is reproducible by this mechanism.

And in the connscale smoke the engine has no children at all — harness/load/failover.py spawns serve with no --shards, and __main__.py runs uvicorn in-process. So every descendant the walk finds is by construction not the engine. A cardinality bound alone would have caught it.

max() then latches the poisoned tick permanently; a mean would have diluted it.

Why #220's gate does not cover it — and why copying that gate would be wrong

My first reading was "handles has no PID-set gate like CPU does." An independent verification pass sharpened it and corrected the remedy, which is the part worth keeping:

#220 hardened the CPU path to degrade set-change intervals to gaps. That is right for a difference — differencing two sums taken over different PID sets is arithmetically invalid however legitimate the joiner. But an instantaneous gauge summed over a genuinely larger subtree is correct. So the fix here is not the same gate. The real gap is that the number carries no record of the set it covered, leaving a legitimate growth and a misresolution indistinguishable — and the SLO draws a verdict anyway.

Secondary, and worse: #220's gate is a change-detector, not a binding check. A step whose first and only resolution adopts a wrong-but-stable subtree yields pa == pb on every interval, so CPU reports too, and the flat-counter guard will not fire because an adopted tree does burn CPU. The smoke profile is exactly that shape.

The exposure was asserted as correct by the test that shipped #220's fix

tests/test_connscale_cpu_probe.py::test_a_membership_changed_interval_is_degraded_to_a_gap ends:

# The non-CPU gauges still read - the process set was observed, only its CPU delta is unsound.
assert d.handles_peak == 61
assert d.working_set_peak_bytes == 6_000_000

Two problems. The rationale is true of the differencing arithmetic and false of a peak over a set never validated to be the engine. And the fixture is physically unrealizable: it varies cpu_pids {100} → {100,200} → {100} while pinning handles=61 across all three ticks, but the Windows probe reads handles and CPU from the same Get-Process rows, so a joining PID necessarily moves both. It constructs the one input shape in which this exposure is invisible, then asserts the pass-through as correct.

Blast radius

The SLO is asserted by the smoke test, so this reds PRs that touch nothing near the harness — it red #298, whose diff was two scripts/asvs/ files.

Fix shape, ranked (proposed, not applied)

1. Validate the walk rather than gate the aggregate. A genuine descendant cannot predate its root. _descendants_windows already queries Win32_Process; add CreationDate and reject any candidate created before the root. /proc/<pid>/stat field 22 gives the same on POSIX. This single change would have prevented the observed number.
2. Record the covering PID set for the FD/RSS sum — on Windows cpu_pids already is that set and is simply not consulted.
3. Give FD/RSS a plausibility guard, the analogue of _CPU_FLAT_GAP_SPAN_S: for a non-sharded engine, a subtree of cardinality > 1 is a wrong binding and should degrade to a gap, not a number.
4. Fix the fixture so handles moves with the PID set; the current one cannot fail on this class.

Not covered by #1101, the sibling SLO defect on the same test — it touches handles_peak once, only to dismiss it as a control for time dilation. Correct about dilation, silent on provenance.