Skip to content

Lock gate-1 thresholds (ratification required) - #33

Merged
MaxGhenis merged 3 commits into
masterfrom
lock-gate1-thresholds
Jul 5, 2026
Merged

Lock gate-1 thresholds (ratification required)#33
MaxGhenis merged 3 commits into
masterfrom
lock-gate1-thresholds

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Locks the gate-1 numbers. Do not merge casually: merging this PR IS the threshold lock — the pre-registration event that must precede the first model run.

Amended twice under adversarial review, both rounds applied before ratification: round 1 (thresholds didn't derive from their stated rule; floor at the wrong scale for the pairs view) and round 2 (the same wrong-scale defect reintroduced on the runs view; clone-guard misattribution; q90 ungated; derivations not machine-checked). Every threshold now derives from a committed deployment-scale floor artifact, and tests/test_gates_derivations.py asserts each locked value equals its stated floor mean + k·sd — the rationale can no longer drift from the numbers.

Geometry, psid_family_earnings_pairs (window 2) — floor = runs/noise_floor_psid_family_ctx20_9822.json (real vs real at ~20%-of-persons scale):

threshold locked floor (mean ± sd) k
C2ST AUC ≤ 0.53 0.5108 ± 0.0046 +4.2
energy ≤ 0.008 0.0034 ± 0.0025 +1.8
coverage ≥ 0.93 0.9577 ± 0.0059 −4.7
q99 ratio ∈ [0.70, 1.40] 1.0428 ± 0.1338 −2.56 / +2.7
q90 ratio ∈ [0.85, 1.15] (new) 1.0064 ± 0.0480 −3.3 / +3.0
w1/sd ≤ 0.06 0.0349 ± 0.0131 +1.9

Geometry, psid_family_earnings_runs (window 3) — floor = runs/noise_floor_psid_family_runs_ctx20_9822.json (deployment scale, per round 2): C2ST ≤ 0.54 (0.5092 + 4.3σ), coverage ≥ 0.90 (0.9507 − 4.0σ). The runs view is a supporting check: at deployment scale a chained-Markov clone straddles its C2ST threshold (0.533–0.542 vs 0.54). The pre-registered guards for chained-model persistence understatement are the battery's autocorrelation and spell tolerances — clone 10yr autocorr 0.207 vs reference 0.539 (4.7× the ±0.07 band) and 4yr 0.533 vs 0.657 (2.1× the ±0.06 band), failing 5/5 seeds.

Battery (absolute bands around battery_reference in runs/noise_floor_psid_family_9822.json): autocorr 2yr ± 0.05, 4yr ± 0.06, 10yr ± 0.07; mobility diagonal ± 0.05 (six origin-diagonal cells, zero bin included); zero persistence ± 0.05 and exit ± 0.05 (one constraint — the identity zero_persistence = 1 − exit_rate is now test-pinned); entry ± 0.02; mean spell length ± 0.25. Change moments (sd/skew/kurtosis), age profiles, and horizon-2 mobility are reported-not-gated.

Pass rule: seed-level conjunction — a seed passes iff every locked threshold on every locked view holds; the gate passes iff ≥4/5 seeds pass geometry and ≥4/5 pass the battery. The candidate refits per seed on the 80% complement and emits trajectories for exactly the holdout persons. Round 2 verified fairness empirically: a faithful truth-process redraw passes 5/5 under the full conjunction; round 1's broken candidates (shuffled dynamics; 18% persistence understatement) fail 5/5.

After merge, flipping locked: true happens in a one-line follow-up before any model run.

🤖 Generated with Claude Code

Geometry thresholds sit at roughly floor mean plus four floor
standard deviations (or twice the floor mean where that is looser),
so sampling noise cannot fail a faithful candidate; battery
tolerances are absolute bands around the committed reference values.
locked stays false until maintainer ratification of this pull
request — the merge is the pre-registration event, and no model runs
before it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
social-security-model Ready Ready Preview, Comment Jul 5, 2026 6:40am

Request Review

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Verdict: AMEND BEFORE RATIFYING.

Verification anchor: from the committed code path (populace_dynamics.harness.{panel,moments} on the family view, filter age 25-59 / periods 1998-2022 / weight>0) I reproduce every geometry floor mean in runs/noise_floor_psid_family_9822.json to 6 decimals and 7 of 8 battery references to float equality. The artifact is sound. The proposed lock on top of it has the following problems.

1. The stated rationale does not produce the proposed numbers. "Floor mean + ~4 floor-sd or 2x floor mean, whichever is looser" yields: c2st <= 0.516 (the 2x-mean branch, 1.010, is vacuous for an AUC), energy <= 0.00707, coverage >= 0.943 (mean - 4 sd), q99 in [0.700, 1.306], w1 <= 0.0369. Proposed vs floor: 0.55 = mean + 16.3 sd; 0.008 = mean + 4.8 sd; 0.90 = mean - 10.5 sd; 1.40 = mean + 5.2 sd; 0.06 = mean + 14.9 sd (3.25x mean). Four of five sit beyond the looser branch; only the q99 lower bound (0.70 = mean - 4.0 sd) follows the rule. A pre-registered rationale that does not derive its own numbers is what any future dispute will cite.

2. The rule is mis-scaled for the protocol anyway. The committed floor is half-vs-half on the full panel (~51k windows/side), but the pass rule scores candidates against 20% holdouts (~20k windows), and c2st plus the tail block run uncapped, so their deployment-scale noise is ~1.6x the committed floor's. Real-vs-real at 20%-vs-20% person scale (same code path, seeds 0-4): c2st 0.5108 +/- 0.0046, energy 0.0034 +/- 0.0025, coverage 0.9577 +/- 0.0059, q99 1.043 +/- 0.134, w1 0.0349 +/- 0.0131. Against that correctly-scaled floor, w1 0.06 (+1.9 sd), q99 1.40 (+2.7 sd), and energy 0.008 (+1.8 sd) are about right — note a naive tightening of w1 to the rule's 0.0369 would sit inside deployment noise — but c2st 0.55 (+8.6 sd) and coverage 0.90 (-9.7 sd) remain far looser than any noise bound.

3. The two loose thresholds demonstrably lack bite. Simulated at protocol scale (4,460 persons, biennial, marginals matched exactly): a fully shuffled candidate (one-step corr 0 vs true 0.73) scores coverage 0.881-0.903 and clears the 0.90 bar on 1/5 seeds; a candidate understating one-step corr by 18% (0.60) scores c2st 0.548-0.557, straddling 0.55 (2/5 clean seeds). At c2st <= 0.53 / coverage >= 0.93 both fail 5/5 seeds while a faithful redraw of the true process passes 5/5 (c2st 0.502-0.507, coverage 0.950-0.972).

4. Window-2 geometry cannot see the failure mode the paper warns about (sec. 5 / Dynamics operator: chained one-period models understate long-spell persistence), and the lock leaves one number guarding it. A chained Markov clone matched to the one-step joint passes all window-2 geometry at the floor by construction — the pairs view is exactly the object such a model fits (sim: c2st 0.504, 5/5 pass) — while its 10yr log-autocorr collapses to 0.730^5 = 0.207 vs reference 0.539. The only locked guard is autocorr_log_10yr +/- 0.07. The artifact already commits two backups the lock omits: autocorr_log_4yr 0.657 (Markov-implied 0.533, gap 0.124 — also catches the clone) and mean_spell_length 2.087 (the only direct long-spell statistic, and the only zero-margin one; entry/exit/zero-persistence are all one-step). Meanwhile the window-3 view registered precisely for this failure mode (psid_earnings_runs) stays unlocked, and it sits on the 3-wave individual product (at most one window per person) when the family panel now supports 81,654 window-3 windows. Family window-3 floor, same code path, seeds 0-4: c2st 0.5071 +/- 0.0049, energy 0.0021 +/- 0.0017, coverage 0.9640 +/- 0.0058; window-3 c2st sees the clone at 0.527-0.536 in the same sim.

5. Battery semantics are underdetermined by the lock. (a) mobility_diagonal_mean 0.604052 reproduces only as the unweighted mean over all six origin-diagonal cells including the zero bin; the equally natural positive-bins reading gives 0.5847 — a 0.019 definitional gap, 39% of the +/-0.05 band. (b) zero_persistence = 1 - exit_rate to 16 digits (0.7010207 = 1 - 0.2989793): one constraint locked twice. (c) Measured perfect-candidate gaps at single-20%-holdout scale already consume much of several bands: mobility -0.020..-0.031 (up to 61% of band, all negative — small-sample attenuation, so candidate panel size must be pinned or the comparison made at matched n), exit/zero-persistence <= 0.0285 (57%), entry <= 0.0077 (38%), autocorr 2yr/10yr 20%/31%. (d) Registered battery families with no committed reference — change sd/skew/kurtosis (the Guvenen higher-order moments the paper names), age profiles, horizon-2 mobility — are silently ungated: declare them reported-not-gated at gate 1 or commit references.

6. The pass rule is exploitable as written. (a) "Every geometry threshold holds on at least 4 of 5 paired seeds": the per-threshold reading passes a candidate that fails a different metric on every seed (0/5 clean seeds). (b) "Pooled holdout" is not an object the harness defines: five independent 20% person-draws overlap (union = 1 - 0.8^5 = 67% of persons, expected multiplicity 1.49) and include persons who were training data for the other seeds' fits. (c) Unstated: that the candidate refits per seed on the 80% complement with the holdout excluded from all fitting and calibration; that split_panel_by_person(fraction=0.2, seed=s) with the picked 20% defines the holdout; that the view filter applies at scoring; and whether the battery compares candidate vs the committed reference (YAML rationale) or vs the holdout battery (pass rule).

Concrete amendments before flipping locked:

item proposed amend to
c2st_auc_max 0.55 0.53 (20%-scale floor mean + 4.2 sd)
prdc_coverage_min 0.90 0.93 (20%-scale floor mean - 4.7 sd)
energy / q99 / w1 0.008 / [0.70, 1.40] / 0.06 keep values; re-derive the YAML rationale from a committed 20%-scale floor artifact
window-3 absent register a family window-3 view, commit its floor, lock c2st <= 0.53 / coverage >= 0.93 on it
battery adds autocorr_log_4yr_tolerance 0.06; mean_spell_length_tolerance 0.25 (holdout-scale gaps <= 0.0195 / <= 0.092)
battery defs pin mobility diagonal (six bins incl. zero, horizon 1); annotate zero_persistence as 1 - exit_rate; battery per seed vs committed reference, pass on >= 4/5 seeds (drop "pooled"); pin candidate panel size
pass rule seed-level conjunction (a seed passes iff all geometry thresholds hold; gate needs >= 4/5 seeds); name the split and scoring calls

The two missing floor artifacts (20%-scale window-2, family window-3) take minutes with the committed pipeline; compute-then-lock beats lock-then-amend.

The review on this pull request found the proposed thresholds did
not derive from their stated rationale and were calibrated at the
wrong scale (full-panel half-vs-half, ~51k windows per side, while
the protocol scores ~20k-window holdouts). This amendment locks
compute-then-lock:

- Two new committed floor artifacts, built by
  scripts/build_gate1_floor_artifacts.py and pinned by reproduction
  tests: the candidate-context window-2 floor (real vs real at
  ~20%-of-persons scale) and the family window-3 runs floor
  (81,654 windows).
- Tighten c2st_auc_max 0.55 -> 0.53 and prdc_coverage_min
  0.90 -> 0.93; keep energy/q99/w1 with rationale re-derived from
  the ctx20 floor. Every threshold now states its floor mean + k*sd
  derivation.
- Lock geometry on a family window-3 runs view (c2st 0.53,
  coverage 0.93): window-2 geometry cannot see chained-model
  persistence understatement.
- Battery: add autocorr_log_4yr (0.06) and mean_spell_length
  (0.25) tolerances; pin the mobility-diagonal definition (zero bin
  included); annotate zero_persistence = 1 - exit_rate as one
  constraint; declare unreferenced families reported-not-gated.
- Protocol: seed-level conjunction pass rule, named split and
  scoring calls, candidate refits per seed on the 80% complement,
  candidate panel pinned to holdout scale; drop the undefined
  "pooled holdout".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Amendment applied in ec70b0a, item by item against the review table:

  1. c2st_auc_max 0.53, prdc_coverage_min 0.93 — locked, derived from the now-committed candidate-context floor (runs/noise_floor_psid_family_ctx20_9822.json, mean + 4.2 sd / mean − 4.7 sd).
  2. energy / q99 / w1 kept, rationale re-derived from the ctx20 artifact with per-metric multipliers stated in the YAML — the rationale now produces its own numbers.
  3. Family window-3 view registered and locked (c2st ≤ 0.53, coverage ≥ 0.93) against runs/noise_floor_psid_family_runs_9822.json (81,654 windows); the individual three-wave runs view is annotated secondary.
  4. Battery adds: autocorr_log_4yr ± 0.06, mean_spell_length ± 0.25.
  5. Battery definitions pinned: mobility diagonal = six origin-diagonal cells including the zero bin (0.6041, vs 0.5847 positive-only, stated in the YAML); zero_persistence annotated as 1 − exit_rate (one constraint); battery scored per seed against the committed reference, ≥4/5 seeds — "pooled holdout" removed; candidate panel pinned to holdout persons; holdout-scale measurement error recorded as accepted at lock. Change moments, age profiles, and horizon-2 mobility declared reported-not-gated.
  6. Pass rule: seed-level conjunction, with the split (split_panel_by_person(panel, "person_id", fraction=0.2, seed=s) after the view filter) and scoring (panel_scorecard(candidate, holdout, view, seed=s)) calls named, and per-seed refits with holdout exclusion stated.

Both new artifacts were rebuilt from the committed pipeline (scripts/build_gate1_floor_artifacts.py) and reproduce the review's numbers exactly; reproduction tests pin them at seed 0 to float precision. Full suite: 79 passed.

Awaiting maintainer ratification — merging remains the lock event.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Verdict: AMEND BEFORE RATIFYING.

Round-2 review of the amended lock (ec70b0a). Reproduction anchor: all 8 stated mean + k·sd derivations reproduce from the committed artifact stats to rounding (script arithmetic on the JSON floors), every round-1 table item is present in gates.yaml with nothing silently dropped, and the 8 locked battery tolerances map 1:1 onto the 8 battery_reference values. The amendment faithfully implemented round 1. But round 1 applied its own scale fix to the pairs view and not to the runs view, and that is the problem.

1. Runs-view thresholds are calibrated at 5× deployment scale — round 1's own medicine, reintroduced. runs/noise_floor_psid_family_runs_9822.json is half_vs_half on the full panel (81,654 windows/side, c2st sd 0.0049), but the protocol scores 20% holdouts. I rebuilt the runs floor at deployment scale (ctx20 construction, ~16k windows/side, seeds 0-4): c2st 0.5092 ± 0.0072, coverage 0.9507 ± 0.0125. Against that, the locked runs c2st_max 0.53 sits at +2.9 sd (not the +4.7 the YAML claims) and runs coverage_min 0.93 at +1.7 sd (not −5.9). These are exactly the too-tight, wrong-scale numbers round 1 rejected for window-2 (comment 4884803805, finding 2). The YAML derivation lines 153-154 are mis-scaled.

2. Fixing the runs scale destroys the runs view's bite — it cannot do both at 20%. I ran round-1's chained-Markov clone (ρ=0.730, the paper's named failure mode) at protocol scale on both views under the seed-level conjunction. It passes all window-2 geometry (pairs c2st 0.501-0.510) and is caught only by runs c2st 0.533-0.542 > 0.53, on 5/5 seeds. But a runs threshold placed ≥4 deployment-sd above noise is ~0.539-0.540, and the clone (min 0.533) then passes 4/5 seeds. So runs c2st = 0.53 fails the clone only because it is 2.9-sd tight. The amendment's own cited clone score "0.527-0.536" (line 168) already straddles 0.53 — internally inconsistent with calling the runs view a guard. The robust clone guard is the battery: autocorr_log_10yr clone 0.207 vs ref 0.539 (gap 0.332 ≫ 0.07 tol) and autocorr_log_4yr 0.533 vs 0.657 (gap 0.124 ≫ 0.06) — >4× over tolerance on 5/5. Amend: stop claiming +4.7/−5.9 sd for the runs view; either (a) re-derive its numbers from a committed deployment-scale runs artifact and label it a supporting check whose bite comes from the autocorr tolerances, or (b) score the runs view half-vs-half deliberately (not 20%) and say so — but then it is off-protocol.

3. Conjunction rule is fair to a faithful candidate — confirmed, no change needed. A truth-process redraw (permanent + AR transitory) passes 5/5 under the seed-level conjunction across both views (pairs c2st ≤ 0.511, runs c2st ≤ 0.512, all tolerances clear). Round-1's broken candidates still fail 5/5 under the amended 0.53/0.93. The k<2 tail thresholds do not clip the faithful redraw. The ≥4/5 rule and the pairs thresholds are sound.

4. Null-mismatch claim does not hold — reporting the negative. Deployment scores the same persons as the holdout (matching age rows); the floor pairs disjoint persons. I scored one identical donor-bootstrap generator both ways: same-persons c2st 0.670 vs disjoint-persons 0.666 (gap +0.004), energy +0.002, coverage −0.002. Age is too coarse a covariate for the same-persons match to make the comparison materially harder, so the disjoint floor is not anti-conservative for c2st/energy/coverage. The pairs 0.53/0.93 null is defensible. (ctx20 seed-overlap is Jaccard 0.11 / 71% reuse; the disjoint-draw c2st sd is larger at 0.0058, so the pairs c2st lock is +3.7 disjoint-sd — still safe — but the committed energy sd 0.0025 is ~8× the independent-draw sd 0.0003, i.e. inflated by the overlapping construction, not a clean noise estimate. Minor.)

5. Two smaller gaps (fresh eyes). (a) q90_ratio is listed in geometry_blocks: tail and the harness computes it, but no threshold locks it — only q99 and w1. A candidate can distort the 90th-percentile ratio (more mass than q99) while passing. Lock it (deployment floor 1.006 ± 0.048 → e.g. [0.85, 1.15]) or add it to reported_not_gated. (b) The reproduction tests pin the artifacts' seed-0 floor values (pytest.approx(values[0])) but nothing asserts 0.53 == round(ctx_mean + 4.2·ctx_sd, 2); the derivation lives only in a comment, so a future artifact rebuild that shifts the floor would silently break the stated rationale without failing a test. Add one assertion binding each locked threshold to its floor+k·sd.

Concrete amendments before flipping locked:

item current amend to
runs c2st_auc_max derivation 0.53 = 0.5071 + 4.7 sd (full-panel floor) commit a deployment-scale runs floor; state 0.53 = 0.5092 + 2.9 sd; label runs a supporting check
runs prdc_coverage_min derivation 0.93 = 0.9640 − 5.9 sd 0.93 = 0.9507 − 1.7 sd at deployment scale (or widen holdout)
clone guard attribution "window-3 view + autocorr" autocorr_log_4yr/10yr are the robust guard; runs c2st is marginal (0.53 catches clone only at 2.9-sd tight)
q90_ratio in tail block, unlocked lock (~[0.85,1.15]) or declare reported_not_gated
derivation test absent assert each locked threshold == floor mean + stated k·sd

Pairs geometry, battery, conjunction rule, and protocol wording are ready. The runs-view scale is the one blocker: it is the exact defect round 1 caught, and the artifact to fix it takes minutes with the committed pipeline.

Round 2 (comment 4885102386) found the runs-view thresholds
calibrated against the full-panel half-vs-half floor — the same
wrong-scale defect round 1 rejected for the pairs view — and showed
the runs view cannot be both scale-honest and clone-catching at
deployment scale.

- Commit a deployment-scale window-3 floor
  (runs/noise_floor_psid_family_runs_ctx20_9822.json) and re-derive
  the runs thresholds from it at the same ~4-sd discipline as every
  other threshold: c2st 0.54 (mean + 4.3 sd), coverage 0.90
  (mean - 4.0 sd).
- Reattribute the chained-clone guard to the battery, where it
  belongs: 10yr autocorr gap 4.7x tolerance, 4yr 2.1x, 5/5 seeds;
  the runs view is labeled a supporting check (the clone straddles
  its threshold at deployment scale).
- Lock q90_ratio_range [0.85, 1.15] on the pairs view (it was
  computed but ungated).
- Make every geometry derivation machine-checkable: a `derivations`
  block per view in gates.yaml plus tests/test_gates_derivations.py
  asserting threshold == floor mean + stated k * sd, that every
  locked threshold has a derivation, that battery tolerances have
  committed references, and the zero_persistence = 1 - exit_rate
  identity.
- Document the ctx20 overlapping-draw construction (disjoint-draw
  c2st sd 0.0058 keeps the pairs lock at +3.3 sd; the committed
  energy sd is ~8x the independent-draw sd, i.e. conservative).
- Restate the pairs q99 lower bound as k = -2.56 (the prior -2.6
  rounds to 0.69, not the locked 0.70).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round-2 amendment applied in d507c22, item by item against the review table:

  1. Runs-view scale — committed the deployment-scale window-3 floor (runs/noise_floor_psid_family_runs_ctx20_9822.json, ctx20 construction, seeds 0-4: c2st 0.5092 ± 0.0072, coverage 0.9507 ± 0.0125 — matching the review's rebuild) and re-derived the runs thresholds from it. Rather than keep 0.53/0.93 with confessed 2.9σ/1.7σ margins, the runs view now follows the same ~4σ discipline as every other threshold: c2st ≤ 0.54 (+4.3σ), coverage ≥ 0.90 (−4.0σ). Option (a) from the review, taken all the way: scale-honest, labeled a supporting check.
  2. Clone-guard attribution — the YAML and PR body now state that the runs view straddles the clone at deployment scale (0.533–0.542 vs 0.54) and that the pre-registered persistence guards are autocorr_log_10yr (clone gap 4.7× tolerance) and autocorr_log_4yr (2.1×), failing 5/5. The internally-inconsistent "0.527–0.536 vs 0.53" guard claim is gone.
  3. q90_ratio — locked on the pairs view: [0.85, 1.15] = 1.0064 − 3.3σ / + 3.0σ.
  4. Derivation binding — every locked geometry threshold now carries a machine-checkable derivations entry ({floor_run, key, k, rounding}) and tests/test_gates_derivations.py (runs everywhere; committed files only) asserts threshold == round(mean + k·sd), that every locked threshold has a derivation, that each battery tolerance has a committed reference, and the zero_persistence = 1 − exit_rate identity. A floor rebuild that shifts any artifact now fails CI instead of silently orphaning the rationale. One restatement it forced: the pairs q99 lower bound k is −2.56, since −2.6 rounds to 0.69, not the locked 0.70.
  5. ctx20 construction note (review finding 4, minor) — the overlapping-draw caveat is documented in the YAML: disjoint-draw c2st sd 0.0058 keeps the pairs lock at +3.3σ; the committed energy sd is ~8× the independent-draw sd, i.e. that margin is conservative.

Findings 3 (conjunction fairness) and 4 (null-mismatch negative) required no change; the faithful-redraw and broken-candidate results are now cited in the YAML rationale as review evidence.

Full suite: 91 passed (was 79; +11 derivation-binding, +1 runs_ctx20 reproduction pin). Awaiting round-3 referee, then maintainer ratification — merging remains the lock event.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Verdict: RATIFY AS-IS. Non-blocking notes in findings 4 and 6.

Round-3 review of amendment d507c22 against round 2 (comment 4885102386). Reproduction anchor: full suite 91 passed locally (79 + 11 derivation-binding + 1 runs_ctx20 pin, as the amendment reply states); CI runs the derivations test on every PR without PSID.

1. Round-2 table lands item-by-item; the diff is exactly scoped. gates.yaml ec70b0a→d507c22 changes only: the runs candidate_context_floor_run, the thresholds comment block, the q90 lock, both derivations blocks, and runs 0.53→0.54 / 0.93→0.90. Battery tolerances, reported_not_gated, and protocol — everything round 2 cleared — are byte-identical; the three prior floor artifacts are untouched; the only new file content is runs/noise_floor_psid_family_runs_ctx20_9822.json, whose stats match round 2's independent rebuild (c2st 0.5092±0.0072, coverage 0.9507±0.0125; 4,443.6 persons/side = protocol scale). Nothing dropped or renamed.

2. All ten derivations reproduce from the committed artifacts at exact float precision. Pairs: 0.5108054+4.2·0.0045535=0.52993→0.53; 0.0033793+1.8·0.0025143=0.007905→0.008; 0.9577148−4.7·0.0059362=0.92981→0.93; q99 1.0428391−2.56·0.1337886=0.70034→0.70 (the restatement was forced: −2.6 gives 0.69494→0.69) and +2.7→1.40407→1.40; q90 −3.3→0.84812→0.85 and +3.0→1.15033→1.15; w1 +1.9→0.05973→0.06. Runs: 0.5091816+4.3·0.0072226=0.54024→0.54; 0.9506836−4.0·0.0125394=0.90053→0.90. Every floor stat quoted in the YAML comments and the PR body matches its artifact at the stated precision, and the +3.3 disjoint-sd note is self-consistent ((0.53−0.5108)/0.0058=3.31), conservative vs round 2's +3.7.

3. The divergence from round 2's literal option (a) — re-deriving the runs view at ~4σ instead of confessing +2.9σ — is the right call, and the loosened view keeps bite. Keep it; do not drop it.

  • The old runs coverage 0.93 fails real-vs-real noise outright: the committed deployment floor's own seed 1 scores 0.9287 < 0.93. A bound that noise violates on 1/5 committed seeds was untenable; 0.90 = mean − 4.0σ is the defensible number.
  • I reran the prior rounds' candidates under the amended locks at protocol scale. Shuffled dynamics: runs c2st 0.799–0.802 and runs coverage 0.788–0.827 — fails both loosened runs thresholds 5/5 (its pairs coverage 0.86–0.91 also fails 0.93 on 5/5, consistent with the cited bite claim). The 18%-understatement candidate: runs c2st 0.594–0.600, fails 5/5. Window-3 joint geometry degrades faster than window-2 under dynamics failures, so the loosening opens no hole for any demonstrated failure mode; the conceded band (runs coverage in [0.90, 0.93) with everything else passing) is substantially noise territory given the floor's observed 0.9287.
  • The clone reproduces exactly as committed: runs c2st {0.533, 0.533, 0.536, 0.538, 0.542} straddling 0.54, failing 1/5 seeds → passes geometry 4/5, caught by the battery — precisely the amendment's stated attribution. Note the runs view is the only geometry that registers the clone at all (a runs-only failure on 1/5 seeds), a supporting signal window-2 structurally cannot produce.
  • Fairness re-verified WITH the new q90 in the conjunction (round 2's fairness run predates the q90 lock): a faithful truth-process redraw passes 5/5 — q90 0.962–1.002 vs [0.85, 1.15], runs c2st ≤ 0.512, runs coverage ≥ 0.951, battery clean.
  • Keep-vs-drop: keep. It is the only locked view spanning two transitions (omnibus, nonlinear, zeros included — not duplicated by the linear log-positive autocorr tolerances), it fails every demonstrated broken candidate 5/5, and at ~4σ it adds no false-failure risk.

4. The derivations test binds, with characterized limits that are acceptable by design. Mutating the runs c2st floor mean by +0.01 (1.4 floor-sd) fails the test. Limits: (a) rounding deadband — a mean drift under half a rounding unit (±0.005 absolute; 0.01–1.1 floor-sd depending on metric) passes silently; any artifact regeneration is itself a public diff, and the PSID-staged reproduction pins catch seed-0 value changes exactly. (b) A post-hoc k edit re-fits a shifted artifact (verified: mean +0.01 with k 4.3→2.88 passes 11/11) — by design: the test enforces threshold↔k↔artifact consistency, while immutability comes from the public-amendment rule in the file header. (c) One uncovered vector: hand-editing an artifact's mean/sd while leaving values[] intact passes every test in every environment (verified: sd faked to 0.02 with mean re-fit, 11/11 pass; the reproduction pins check values[0] only). The committed values[] arrays still expose such an edit to trivial recomputation, so this is a CI-detectability gap, not an evidentiary hole. Cheap hardening, fine post-merge: assert mean == mean(values) and sd == stdev(values, ddof=1) in test_gates_derivations.py.

5. Clone-guard reattribution verified. battery_reference: ac2yr 0.730124, ac4yr 0.657167, ac10yr 0.538879. Markov-implied clone: 0.730^5 = 0.207, gap 0.332 = 4.7× the 0.07 band; 0.730^2 = 0.533, gap 0.124 = 2.1× the 0.06 band — both as stated. Empirical rerun: ac4yr fails 5/5 (clone 0.536–0.546), ac10yr fails 5/5 (0.203–0.234). No remaining text claims the runs view catches the clone: thresholds block, PR body, and commit message all state straddle-plus-battery.

6. Non-blocking notes. (a) The views comment (gates.yaml:43–49) still motivates the runs lock with "because window-2 geometry cannot see chained-model persistence understatement" — literally true of window-2, but a hostile reader could juxtapose it against the thresholds block's "SUPPORTING check, not the guard"; one clarifying clause closes it, and the dated, review-citing thresholds block controls the numbers either way. (b) Remaining computed-but-undeclared quantities: the runs-view scorecard also emits energy/q99/q90/w1 (present in the runs_ctx20 artifact) with no lock and no reported_not_gated line — both prior rounds prescribed runs = {c2st, coverage}, and two-step marginals follow from correct one-step conditionals, so this is declaration-only; likewise change_moments registers mean but the declaration lists only sd/skew/kurtosis, and transition_rates (marital status, gate-2 domain) carries neither tolerance nor declaration. None create pass/fail ambiguity — the pass rule quantifies over locked thresholds and tolerances only. (c) The PR body's "the rationale can no longer drift from the numbers" slightly overstates 4(a)–(c); the operative protection is the public-amendment rule, which gates.yaml itself states.

Every locked number derives from committed evidence and is machine-checked; the amendment fixed what round 2 found without disturbing anything round 2 cleared. Ratify by merging; flip locked in the follow-up as documented.

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Third-round review (independent machine): ratification-ready.

  • Hand-checked every pairs-view derivation against the committed floor artifact: all seven reproduce the locked values under the stated rule (e.g. C2ST 0.5108 + 4.2×0.0046 = 0.5301 → 0.53; q99 lower 1.0428 − 2.56×0.1338 = 0.7003 → 0.70; w1/sd 0.0349 + 1.9×0.0131 = 0.0598 → 0.06).
  • tests/test_gates_derivations.py: 11/11 pass locally — the machine-check that keeps rationale and numbers from drifting is real.
  • The runs-view disclosure (chained-Markov clone straddling its C2ST threshold at deployment scale) is honest and the compensating autocorrelation/spell guards are quantified with the clone failing 5/5 — the empirical fairness check in both directions is exactly what a pre-registration should show.

Nothing further from review. Per the PR's own terms, merging is the lock — that signature belongs to the ratifier, not the reviewer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant