Dense arm SSI reconcile exceeds the #431 runaway cap (11.4% vs 10%) — instrument per-pass deltas, then adjudicate

[MaxGhenis]

What happened

The Build M dense arm (populace-us-2024-buildm-dense-c48f7da-20260716T233015Z, launched as joint-certification evidence while the sparse arm was blocked on #445) failed after 3 SSI take-up reconciliation passes:

RuntimeError: SSI take-up reconciliation did not remain count-faithful on returned weights after 3 pass(es):
SSI take-up swap delta 749467.324 exceeds the runaway sanity cap 658513.070

• Dense equilibrium delta: 749,467 = 11.38% of the fresh national (≈6,585,131).
• The Gate the SSI swap delta at a runaway cap, not assignment granularity #431 cap (0.10 × fresh national) was calibrated on sparse behavior: attempt 9 recorded 518,639 = 7.96% (cap 651,248), within bound, and every sparse attempt since has passed this gate.
• No per-pass delta telemetry survives the raise — the reconcile aborts before any diagnostics artifact is written, so the pass-by-pass trajectory (converging vs oscillating vs flat) is unknowable post-hoc. That is its own small defect: the raise should carry the per-pass deltas it observed.

Reading

The #431 adjudication established the delta is the solve's equilibrium residual on the SSI family (driven by the #424 candidate undercount: 4.03M candidates vs 7.4M official recipients — the assignment cannot reach official counts, so each refit moves recipient mass), not assignment noise. The dense frame (full 865k pool, ratio-bounded weights, no L0 selection) evidently reaches its equilibrium further from the pre-refit flags than the sparse frame does: 11.4% vs 8.0%. Whether 11.4% is the dense arm's honest equilibrium (cap mis-sized for dense) or a genuine runaway (dense reconcile diverging) cannot be distinguished without the per-pass trajectory.

Options

1. Instrument first (recommended): make the reconcile's terminal raise (and diagnostics on success) carry per-pass deltas; rerun the dense arm once (its checkpoint is warm — the failed run wrote it, so the rerun is solve+reconcile only). Converging-but-over-cap ⇒ adjudicate a dense-specific cap with the measured equilibrium as justification, the Gate the SSI swap delta at a runaway cap, not assignment granularity #431 pattern. Oscillating/diverging ⇒ real defect, diagnose.
2. SSI candidate undercount: modeled eligible mass far below SSA recipient counts #424 root fix (candidate-universe restoration, architecture week) — shrinks the equilibrium residual on both arms; the durable fix either way.
3. Certify sparse without dense evidence this cycle — precedent question: Build J shipped joint sparse+dense; dropping dense weakens the verdict evidence and needs Max's explicit call.

The sparse arm (the actual release artifact) is unaffected: its SSI reconcile has passed the cap on every attempt since #431, and attempt 16 (with the #446 keogh mass protection) is running now.

NOT proposing a cap bump without the trajectory measurement — that would be adjusting a gate to pass the data we have (#431 exists because a mis-specified bound burned attempt 8).

[MaxGhenis]

Ops finding from the dense-arm GCP contingency runs (while chasing this issue's trajectory evidence): the dense builder's resident set scales with thread/worker fan-out, not a fixed peak — measured anon-RSS at OOM: 125GB on 16 vCPU/128GB, then 249GB on 32 vCPU/256GB (same commit af02c91, same inputs; linux OOM-killer both times, no co-tenants). Same code on the 128GB Mac peaks ~60-85GB resident only because macOS compresses ~2:1 and swaps. This also explains the local jetsam-magnet behavior across four local dense attempts. Mitigation now running: OMP/MKL/OPENBLAS/NUMEXPR_NUM_THREADS=16 + LOKY_MAX_CPU_COUNT=16 on the 256GB box (expected ~125GB-class peak, 2x headroom). Follow-up worth filing if it proves out: the release tool should set bounded default thread pools itself instead of inheriting machine geometry.

[MaxGhenis]

Trajectory evidence is in (#448 telemetry, local dense run 4 at af02c91, full-pool arm):

pass 1: delta=784,287.697  cap=655,242.471  (11.97%)
pass 2: delta=764,606.628  cap=657,079.084  (11.64%)
pass 3: delta=749,467.324  cap=658,513.070  (11.38%)

Verdict: converging, not oscillating — monotone decreasing every pass. This is the converging-but-over-cap arm of the adjudication. But the decrements decelerate (−19.7k, −15.1k); a geometric extrapolation puts the dense equilibrium near ~696k ≈ 10.6% of fresh national — above the 10% cap — so raising SSI_TAKE_UP_RECONCILIATION_MAX_PASSES (a bare constant, not CLI-exposed) is unlikely to clear the gate alone.

Recommendation (maintainer's call — this is a gate change): a dense-arm-specific runaway cap at 0.12 × fresh national, the #431 pattern verbatim — the delta stays recorded in the manifest with this trajectory as the sizing justification; the cap still catches a solve that abandons the family (a genuine runaway would not walk monotonically downward). Sparse keeps 10% (its equilibrium measured 7.1–8.0% across attempts 9–16). The durable fix remains #424 (candidate-universe restoration), which shrinks the equilibrium on both arms and lets the caps converge back to one number.

Ops note: both cloud OOM kills and the thread-cap mitigation are documented above; the capped 256GB VM run was stopped after this trajectory landed (no value walking into the known wall) — warm checkpoint retained for relaunch on whatever the maintainer decides.