Gate-1 amendment proposal: decision-relevant recalibration

[MaxGhenis]

What this is: an amendment OBJECT, not a gate change

This PR is the gate-1 amendment proposal under the locked pre-registration contract. It proposes changes and changes nothing. The locked gate_1.thresholds block is untouched — locked: true stays and every locked value is byte-identical (git diff on gates.yaml is a pure addition, zero deletions). No model reads the new amendment_proposed subsection; the harness scores only the locked block.

The process, explicitly:

1. This proposal is inert. It decides nothing on merge-as-is.
2. Adopting it requires a fresh adversarial referee round run against this proposal, per the contract's amendment rule ("locked thresholds change only through a public amendment noted in this file's history" + a fresh referee round).
3. Ratification = maintainer merge of the follow-up that flips the proposed lines live — exactly as PR Lock gate-1 thresholds (ratification required) #33/Flip gate-1 to locked following ratification #39 did for the original lock.
4. Until that ratifying merge, nothing changes.

Do not merge this as a gate change. It is the object a referee round deliberates over.

Evidence chain (all reported-not-gated; none touched a gate)

• C2ST forensics: what the classifier reads at 0.547 #54 (C2ST forensics): at the locked pairs-view c2st ~0.547 the two best candidates fail through structurally different defects at the same AUC; the classifier reads a window-3 texture coordinate, not a benefit-space divergence.
• Gate-1 candidate 7: k-NN conditional rank bootstrap #55 (candidate 7, the ninth run): FAIL (geometry 2/5, battery 2/5). Best classifier scores of any candidate — first to pull pairs-view c2st under threshold on all five seeds — but the runs-view c2st and 10-year autocorrelation each clip 3/5.
• Downstream relevance: what the classifier residual costs in benefit space #56 (downstream relevance): the candidate-7 residual mapped into PIA-proxy benefit space. Central benefit gaps are small (mean +1.85%, median +1.02%, both well inside ±5%) and the weighted KS (0.025) sits inside the 0.028 real-vs-real noise floor. The one material miss is the Q0 zero-anchor-earnings subgroup at +9.30% — the corner the progressive 415(a) formula (90% replacement below the first bend point) weights most.
• Candidate 2 design: latent-permanent conditioned chained QRF (research memo) #42: the model registry where each candidate's spec is frozen before its one-shot run.

The new committed anchor

runs/pia_proxy_floor_9822.json (built by scripts/build_pia_proxy_floor.py): the real-vs-real PIA-proxy distribution floor at deployment scale. Per seed s in 0–4, the ctx20 construction (40% person draw at seed 1000+s, halved person-disjointly at seed s) on the full locked filtered panel (22,300 persons — not a train split), both disjoint halves pushed through the pinned PIA-proxy functional imported verbatim from the merged scripts/build_downstream_relevance.py (single source of truth: wage-base cap, NAWI-to-2022 indexing, top-min(10,n) mean over observed positives, 2022 415(a)/415(g) PIA; zero-positive persons at 0; anchor-period weights).

Headline floor (mean ± sd across seeds), pe-us pinned at bf71be3b:

Statistic	Real-vs-real floor
|mean % gap|	1.86 ± 0.76
|median % gap|	1.61 ± 0.99
weighted KS	0.0288 ± 0.0078
Q0 (zero-anchor) mean % gap	pooled 2.66% (per-seed 5.25 ± 4.53)

The floor is real-vs-real, so no candidate is generated and the oracle import path does not pull populace.fit — the reproduction test (tests/test_pia_proxy_floor.py, seed-0, pinned to float precision) needs only PSID and runs live in the repo .venv; it also asserts populace.fit never enters sys.modules.

The two proposed changes

Change 1 — runs-view c2st_auc_max: gated → reported-not-gated. The window-3 runs-view c2st is the single threshold candidate 7 clips (3/5 seeds), yet #56 shows that residual is benefit-immaterial (KS inside the noise floor; central gaps ≤1.9%) and #54 attributes it to a window-3 texture coordinate that does not reach benefits. The runs view was already annotated in the lock as a supporting check (a chained one-period clone straddles it); the pre-registered guard for chained-model persistence understatement is the battery's autocorrelation and spell tolerances, which stay gated and fail such clones 5/5. The pairs-view c2st stays gated.

Change 2 — a NEW gated benefit_space block on the candidate PIA-proxy panels per seed:

• abs(mean % gap) ≤ 5, abs(median % gap) ≤ 5 — a priori, per seed
• decile % gap ≤ 5 for d3–d9 — a priori, per seed. d1 and d2 reported-not-gated: both are denominator-fragile (bottom PIA-proxy ~$290–500/mo, so ~$80–145 absolute gaps read as huge percents). The new anchor shows the real-vs-real floor itself clips ±5% on 5/5 seeds at d1 and 3/5 at d2 (decile_seeds_clipping_5pct) — gating either would reject reality. Downstream relevance: what the classifier residual costs in benefit space #56 already flagged d1; the new anchor extends the same finding to d2.
• weighted KS ≤ committed anchor mean + 4×sd = 0.0599 — the one floor-derived band, with a machine-checkable derivations entry (mirrors the locked geometry derivations).
• abs(Q0 zero-anchor-subgroup mean % gap) ≤ 5, scored on the pooled across-seed magnitude (the per-seed Q0 floor is noisy — ~900 persons/side, reaching ~12.8% on one seed — so the across-seed mean is the decision-relevant level; it cancels that noise, real is unbiased at |pooled| 2.66%).

The ±5% bands cite the paper's pre-stated ±5%-of-SSA-statistics success criterion ("AIME distribution for retired workers … within 5 percent on key percentiles", Phase-1 acceptance table in docs/operationalizing-longitudinal-construction.md and docs/evaluation-and-model-selection.md; the paper carries the same standard) as their a priori source. The PIA-proxy is a monotone statute-shaped transform of the AIME-proxy, so its gaps map onto that criterion.

Rationale — recalibration toward decision-relevance, NOT relaxation. Candidate 7 measured +9.3% on Q0, so this block adds strictness exactly where the evidence says the residual bites. Net strictness where it matters increases: change 1 removes one benefit-immaterial texture gate, change 2 adds a benefit-material subgroup gate.

Pass rule. Seed-level conjunction folded into the existing geometry conjunction: a seed passes benefit_space iff every per-seed gated metric holds at that seed, and the block passes iff ≥4/5 seeds pass the per-seed metrics and the pooled Q0 gate holds.

Candidate 7 would still fail

Under this proposed block, candidate 7 FAILS: it passes the seed-level central/decile/KS gates on all five seeds (its benefit-space central gaps and KS are inside the noise floor), but its pooled Q0 gap is +9.30% > 5. Adopting the amendment does not retroactively pass the ninth run.

Discriminator check on committed evidence (runs/pia_proxy_floor_9822.json vs runs/downstream_relevance_c7_v1.json): real-vs-real PASSES the block (seed-level 5/5; pooled |Q0| 2.66% ≤ 5) and candidate 7 FAILS it (pooled |Q0| 9.30% > 5). A proposed gate that passes reality and fails the ninth run is a valid discriminator — a fresh referee round must confirm this.

Reproducibility (the proposal's own numbers must hold)

• tests/test_gates_derivations.py binds the proposed block: the KS band equals the committed anchor mean + stated k×sd; every proposed gated metric carries a derivation or an a-priori-source citation; the anchor's pooled stats recompute from its stored per-seed values; and the reported-not-gated d1/d2 partition matches the anchor's per-decile fragility.
• tests/test_pia_proxy_floor.py: always-runnable internal-consistency (recompute-from-per-seed, schema, headline KS block) plus the seed-0 reproduction pin (skipif PSID absent; no importorskip populace.fit).
• ruff + black -l 79 clean. Full pytest green in the repo .venv (275 passed, 9 skipped — the 9 skips are pre-existing populace.fit gate reproductions, unchanged by this PR).

Run the anchor:

export POPULACE_DYNAMICS_PE_US_DIR=$HOME/PolicyEngine/policyengine-us-main
.venv/bin/python scripts/build_pia_proxy_floor.py

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
social-security-model	Ready	Preview, Comment	Jul 6, 2026 10:21pm

[MaxGhenis]

Verdict: AMEND BEFORE RATIFYING. The quantitative design checks out end to end — anchor reproduces bit-identically, every stated derivation holds, the block separates reality from two structurally different broken candidates, the proposal is purely additive and inert. Two things must change in the amendment object before a referee round ratifies it: the a-priori citation is misattributed (finding 1), and change 1's retroactive footprint is undisclosed (finding 2). Both are text fixes; no numeric or code changes needed.

Verified (independently recomputed):

• Purely additive: 0 deletions; gates.yaml == origin/master + inserted block; locked values byte-identical. Inert: every runner reads gates.gate_1.thresholds only (load_gate1_thresholds); nothing outside the new tests reads amendment_proposed.
• KS band: 0.0288372 + 4×0.0077679 = 0.0599089 → 0.0599. Derivation tests bind (a value edit fails pytest.approx; a k edit is a visible diff to the stated rule).
• Anchor reproduces: full 5-seed in-memory rebuild of pia_proxy_floor_9822.json matches every float to 1e-9 on pe-us bf71be3b (revision recorded in-artifact and asserted by the repro test before comparing floats).
• Pooled-Q0 design: per-seed real-vs-real |Q0| = [2.74, 5.42, 4.22, 12.80, 1.06] — a per-seed 5% gate rejects reality on 2/5 seeds (3/5 pass < 4), so pooled is the only 5%-level design that passes reality (|pooled| 2.66) and fails c7 (+9.30, verified from downstream_relevance_c7_v1.json; c7 passes the per-seed central/decile/KS gates 5/5 as claimed).
• d1/d2 partition binds against the anchor: clip counts d1 5/5, d2 3/5 (≥ majority 3), d3–d9 0/5; the test forces re-partition on any rebuild drift.
• Suite: 275 passed, 9 skipped in .venv (skips = populace-fit gate reproductions), ruff + black clean — as stated.

1. (blocking) The ±5% a-priori citation is misattributed — fix the text, keep the number. paper/paper.qmd contains no ±5% criterion (grep "5 percent", "±5", "within 5": zero hits); "the paper carries the same standard" is false, and CLAUDE.md's "±5% of SSA statistics" is a writing-style example, not a registration. The real pre-stated sources are the two docs' acceptance tables, and both hedge: "The exact tolerances should be refined during implementation" (evaluation-and-model-selection.md), "not that every number above is final" (operationalizing-longitudinal-construction.md), and their object is model-vs-SSA-benchmarks for retired workers' AIME percentiles, not candidate-vs-holdout gaps on a prime-age top-10 proxy. Also, "monotone transform, so its gaps map onto that criterion" conflates percentiles with percent gaps: the concave 415(a) formula compresses percent gaps by elasticity 0.32·AIME/PIA ≈ 0.54 at the median AIME-proxy (~$2,150), ~0.73 at d9, 1.0 below the first bend — so a 5% PIA-proxy band is equivalent to roughly a 7–9% AIME band over d4–d9, i.e. looser than the cited criterion in its own units. Fix in this PR: attribute to "the docs' draft Phase-1 acceptance tables, adopted here as a fixed band in benefit units (the decision-relevant object)", state the concavity dilution, and drop the paper/CLAUDE.md attributions. The band itself is fine — pre-stated magnitude, reality clears it 1.9× pooled on Q0 — but the first amendment must not set the precedent that a-priori citations go unchecked. (Same misattribution sits in the merged SUCCESS_CRITERION_TEXT and the anchor's success_criterion.text; fixing gates.yaml is the binding part.)

2. (blocking, disclosure) Change 1 flips four committed geometry verdicts — say so. Recomputing every committed run's per-seed checks with runs-view c2st removed: baseline 0/5 → 5/5, latent_perm v1 3/5 → 5/5, v2 3/5 → 4/5, candidate 7 2/5 → 5/5 (rank_kernel, rank, splice v1/v2 stay 0/5 via pairs-view). No overall verdict flips — battery fails all four (baseline 0/5 on 4yr+10yr autocorr; latent_perm 0/5; c7 2/5) — but the rationale's "the single threshold on which candidate 7 clips" understates that runs-c2st was also the only geometry catch for the baseline and latent-perm family. Post-amendment, the guard against that family concentrates in autocorr_log_10yr (one number, tolerance 0.07; c7's passing seed sits at deviation 0.0688) plus pooled Q0. Both fired for every real instance, so the demotion is defensible — but #56 validated one candidate's residual as benefit-immaterial, not every defect expressible at runs-c2st ~0.55 (#54 itself shows different defects at the same AUC), and a trivariate warp preserving pairs joints, the three gated autocorr lags, and h1 mobility would post-amendment be gated by nothing (h2 mobility, change sd/skew/kurt, spell-length distribution are all reported-not-gated; the top-10 proxy is order-invariant). Fix: disclose the flip table in the amendment text, and consider a tripwire: "a candidate passing all gates while runs-view c2st exceeds the old 0.54 on ≥3/5 seeds triggers a mandatory downstream-relevance analysis (per #56) before certification." That keeps the demotion honest without re-gating texture.

3. (supports the amendment — new evidence) A prior broken candidate also fails the block. I regenerated the registered baseline QRF per seed (byte-identical run_gate1_baseline machinery, ~17s/seed) and scored the proposed block: pooled |Q0| = 19.01 (per-seed +23.2/+17.7/+21.9/+11.7/+20.6, all same-signed), per-seed metrics 4/5 (seed 1 clips d4 at +5.15). So the block fails reality never, c7 (9.30), and the baseline (19.01) — it is not c7-bespoke, and the Q0 gate has real teeth against the chained family (backcasts systematically inflate zero-anchor histories). Two calibration notes worth carrying into the text: (a) only Q0 ever binds — the KS/central/decile bands are calibrated on disjoint-person noise but scored on paired same-person panels with shared real anchor rows, so even broken candidates score KS ≤ 0.049 vs the 0.0599 band; harmless, but the block's teeth are the pooled Q0 gate, not the seed-level bands. (b) For the same reason, "c7's KS 0.025 sits inside the 0.028 noise floor" and "central gaps inside the noise floor" use the wrong null for paired statistics — c7's +1.85% mean is same-signed on 5/5 seeds (systematic bias, just small vs the band). Right conclusion, wrong framing; state magnitude-vs-band, not vs the disjoint floor.

4. (non-blocking) Two cheap hardenings for the referee round. (a) Pooled-signed Q0 admits sign-cancellation: a candidate with per-seed Q0 biases of ±12% alternating by seed passes at |pooled| ≈ 0. Exploiting it requires split-seed-conditioned behavior (auditable in the frozen spec; both observed broken candidates are same-signed), but a per-seed |Q0| ≤ 15 cap inside the per-seed conjunction is reality-safe (floor max 12.80; 4/5 rule absorbs noise) and bounds the vector. (b) d1/d2: the %-fragility evidence is honest (gating either rejects reality), but the stated cause is the denominator — so gate the numerator: absolute-dollar bands at floor mean+4sd ≈ $240 (d1) / $180 (d2) pass reality by construction and catch a candidate grossly wrong at the bottom, which is currently ungated below d3.

5. (non-blocking, functional notes) The weighted KS is invariant under the strictly monotone statute layer, so the "benefit-space" KS gate is identically the AIME-proxy (earnings-functional) KS — the ×12×2 scale cancels exactly there. For the percent bands the constant is applied identically to both sides, but it sets bracket placement (÷24 puts the median AIME-proxy at ~$2,150, deep in the 32% bracket), and with it how strict "5%" is in earnings units — worth one sentence so nobody reads statute-specificity into the KS gate. The top-10 functional is order- and below-top-10-invariant: the block cannot see timing/sequence defects and must never be argued as a substitute for the remaining gated views/battery (the amendment doesn't, but the precedent should say it).

6. (non-blocking, flip-PR mechanics + nits) The ratifying PR must remove runs-view c2st_auc_max from geometry and derivations.rules together (test_every_locked_geometry_threshold_has_a_derivation asserts set equality) and add it to reported_not_gated. Builder docstring says gate_1.thresholds.amendment_proposed; the block is a sibling: gate_1.amendment_proposed. And to keep the spec-freeze ethic clean, the ratifying PR should state the ordering explicitly: the amended thresholds lock before the next candidate registers on #42.

Process is otherwise exactly right for a first amendment: inert proposal object, machine-checked derivations, evidence chain in committed artifacts, ratification path matching the #33/#39 precedent.

[MaxGhenis]

Referee round applied in d02cdf5, item by item:

1. The ±5% misattribution — fixed with exact provenance. The a_priori_source no longer claims the paper (which carries no such criterion). It now states: the 5% figure originates in the DRAFT Phase-1 acceptance tables (named files), in AIME units; this amendment ADOPTS ±5% as its own proposal-time criterion for PIA-proxy space; and PIA concavity makes a 5% PIA-space band STRICTER than the draft tables' AIME-space figure (≈7–9% AIME-equivalent at the 2022 bend points) — the referee's own arithmetic, now in the text. The source key renames paper_5pct_criterion → amendment_5pct_criterion. No numeric change.
2. The demotion disclosure — added in full. A demotion_disclosure field lists every committed-run geometry-verdict flip (baseline 0/5→5/5, candidate 2 3/5→5/5, candidate 3 4/5, candidate 7 2/5→5/5), states that NO overall gate-1 verdict changes (each still fails the battery, and the benefit-space block — including the referee's regenerated baseline at pooled Q0 +19.0 vs reality's 2.66), and acknowledges the guard concentration: after the demotion, the anti-persistence duty rests on the battery autocorrelation bands plus pooled Q0 — the battery ladder having failed every persistence-broken candidate 5/5 across all nine committed runs.

Also folded in (test hygiene, referee-adjacent): the anchor reproduction test now SKIPS with both revisions named on a mismatched pe-us checkout, per the PR #41 precedent, instead of failing.

Everything the referee verified clean is untouched: the anchor artifact, the KS derivation (0.0288 + 4×0.0078 = 0.0599), the pooled-Q0 design, the d1/d2 partition, and every locked value (byte-identical). Derivation tests: 21 passed locally; live reproduction passes against the pinned checkout.

Awaiting a verification pass on the two fixes, then maintainer ratification — merging this PR is the amendment's ratification event.

[MaxGhenis]

Verdict: RATIFY AS-IS. Both findings from my review are resolved in d02cdf5, verified against committed artifacts; nothing else regressed.

Finding 1 (a-priori misattribution) — resolved. a_priori_source now states exact draft-table provenance in AIME units, explicitly disclaims the paper ("the paper itself carries no such criterion — corrected per the amendment referee round"), adopts ±5% as this amendment's own proposal-time criterion (2026-07-06, before any candidate scored against it), and carries the concavity arithmetic I gave (5% in PIA-proxy space ≈ 7–9% AIME-equivalent at the 2022 bend points, i.e. stricter than the draft tables' 5%). Source key renamed paper_5pct_criterion → amendment_5pct_criterion on all four gated metrics; the CLAUDE.md attribution is dropped. This is the honest sourcing I asked for.

Finding 2 (demotion disclosure) — resolved, every claim re-verified. Recomputing each committed run's per-seed geometry with runs-view c2st removed reproduces the demotion_disclosure block exactly:

• baseline (Gate-1 baseline run: chained weighted QRF #40) 0/5 → 5/5; cand 2 / latent_perm (Gate-1 candidate 2: latent-permanent conditioned chained QRF #43) 3/5 → 5/5; cand 3 / pa-decomp (Gate-1 candidate 3: persistence-aware permanent-component decomposition #44) 3/5 → 4/5 amended (seed 3 also fails pairs c2st, which stays gated — as stated); cand 7 (Gate-1 candidate 7: k-NN conditional rank bootstrap #55) 2/5 → 5/5.
• "NO overall verdict flips to pass": all four gate_1_pass=False (baseline/c2/c3 fail autocorr 5/5; c7 fails 10-yr 3/5 + battery 2/5). Confirmed.
• Guard-concentration claim: baseline/c2/c3 each fail an autocorrelation rung on 5/5 seeds — the battery ladder does carry the anti-persistence duty for every same-signed persistence-broken candidate. Confirmed.
• Referee-verified baseline pooled Q0 +19.0 matches my independent regeneration (19.01); c7 +9.3 and reality 2.66 confirmed.

Invariants still hold on the new commit:

• Locked block byte-identical to origin/master (lines 1–288 diff-clean); gates.yaml still additive-only (0 deletions over origin/master).
• No scoring-time reader touches amendment_proposed / demotion_disclosure / a_priori_source (only a builder docstring mentions the path).
• Derivations still bind: tests/test_gates_derivations.py + tests/test_pia_proxy_floor.py = 22 passed on pe-us bf71be3b, seed-0 reproduction included (ran, not skipped).
• Full suite 275 passed, 9 skipped (unchanged) — the two-file change is inert.

On the test-hygiene change (non-blocking, fine): turning the pin-mismatch assert into pytest.skip (PR #41 precedent) does not weaken the anchor guarantee — on the pinned revision the reproduction still runs and pins to 1e-9 (verified); off-pin it skips instead of failing CI on an environment mismatch, which is correct for a repro that legitimately can't run off-pin.

Two cosmetic nits remain, neither blocking and both deferrable to the ratifying flip-PR: the builder docstring still says gate_1.thresholds.amendment_proposed (block is the sibling gate_1.amendment_proposed), and the flip-PR must move runs-view c2st_auc_max out of both geometry and derivations.rules together (set-equality test) while adding it to reported_not_gated. My non-blocking hardening suggestions (per-seed |Q0| cap against sign-cancellation; absolute-dollar d1/d2 bands) remain optional inputs for the referee round, not ratification blockers.

Clean amendment object. Ship it to the referee round.