Lock gate-1 thresholds (ratification required) - #33
Conversation
Geometry thresholds sit at roughly floor mean plus four floor standard deviations (or twice the floor mean where that is looser), so sampling noise cannot fail a faithful candidate; battery tolerances are absolute bands around the committed reference values. locked stays false until maintainer ratification of this pull request — the merge is the pre-registration event, and no model runs before it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Verdict: AMEND BEFORE RATIFYING. Verification anchor: from the committed code path ( 1. The stated rationale does not produce the proposed numbers. "Floor mean + ~4 floor-sd or 2x floor mean, whichever is looser" yields: c2st <= 0.516 (the 2x-mean branch, 1.010, is vacuous for an AUC), energy <= 0.00707, coverage >= 0.943 (mean - 4 sd), q99 in [0.700, 1.306], w1 <= 0.0369. Proposed vs floor: 0.55 = mean + 16.3 sd; 0.008 = mean + 4.8 sd; 0.90 = mean - 10.5 sd; 1.40 = mean + 5.2 sd; 0.06 = mean + 14.9 sd (3.25x mean). Four of five sit beyond the looser branch; only the q99 lower bound (0.70 = mean - 4.0 sd) follows the rule. A pre-registered rationale that does not derive its own numbers is what any future dispute will cite. 2. The rule is mis-scaled for the protocol anyway. The committed floor is half-vs-half on the full panel (~51k windows/side), but the pass rule scores candidates against 20% holdouts (~20k windows), and c2st plus the tail block run uncapped, so their deployment-scale noise is ~1.6x the committed floor's. Real-vs-real at 20%-vs-20% person scale (same code path, seeds 0-4): c2st 0.5108 +/- 0.0046, energy 0.0034 +/- 0.0025, coverage 0.9577 +/- 0.0059, q99 1.043 +/- 0.134, w1 0.0349 +/- 0.0131. Against that correctly-scaled floor, w1 0.06 (+1.9 sd), q99 1.40 (+2.7 sd), and energy 0.008 (+1.8 sd) are about right — note a naive tightening of w1 to the rule's 0.0369 would sit inside deployment noise — but c2st 0.55 (+8.6 sd) and coverage 0.90 (-9.7 sd) remain far looser than any noise bound. 3. The two loose thresholds demonstrably lack bite. Simulated at protocol scale (4,460 persons, biennial, marginals matched exactly): a fully shuffled candidate (one-step corr 0 vs true 0.73) scores coverage 0.881-0.903 and clears the 0.90 bar on 1/5 seeds; a candidate understating one-step corr by 18% (0.60) scores c2st 0.548-0.557, straddling 0.55 (2/5 clean seeds). At c2st <= 0.53 / coverage >= 0.93 both fail 5/5 seeds while a faithful redraw of the true process passes 5/5 (c2st 0.502-0.507, coverage 0.950-0.972). 4. Window-2 geometry cannot see the failure mode the paper warns about (sec. 5 / Dynamics operator: chained one-period models understate long-spell persistence), and the lock leaves one number guarding it. A chained Markov clone matched to the one-step joint passes all window-2 geometry at the floor by construction — the pairs view is exactly the object such a model fits (sim: c2st 0.504, 5/5 pass) — while its 10yr log-autocorr collapses to 0.730^5 = 0.207 vs reference 0.539. The only locked guard is autocorr_log_10yr +/- 0.07. The artifact already commits two backups the lock omits: autocorr_log_4yr 0.657 (Markov-implied 0.533, gap 0.124 — also catches the clone) and mean_spell_length 2.087 (the only direct long-spell statistic, and the only zero-margin one; entry/exit/zero-persistence are all one-step). Meanwhile the window-3 view registered precisely for this failure mode (psid_earnings_runs) stays unlocked, and it sits on the 3-wave individual product (at most one window per person) when the family panel now supports 81,654 window-3 windows. Family window-3 floor, same code path, seeds 0-4: c2st 0.5071 +/- 0.0049, energy 0.0021 +/- 0.0017, coverage 0.9640 +/- 0.0058; window-3 c2st sees the clone at 0.527-0.536 in the same sim. 5. Battery semantics are underdetermined by the lock. (a) mobility_diagonal_mean 0.604052 reproduces only as the unweighted mean over all six origin-diagonal cells including the zero bin; the equally natural positive-bins reading gives 0.5847 — a 0.019 definitional gap, 39% of the +/-0.05 band. (b) zero_persistence = 1 - exit_rate to 16 digits (0.7010207 = 1 - 0.2989793): one constraint locked twice. (c) Measured perfect-candidate gaps at single-20%-holdout scale already consume much of several bands: mobility -0.020..-0.031 (up to 61% of band, all negative — small-sample attenuation, so candidate panel size must be pinned or the comparison made at matched n), exit/zero-persistence <= 0.0285 (57%), entry <= 0.0077 (38%), autocorr 2yr/10yr 20%/31%. (d) Registered battery families with no committed reference — change sd/skew/kurtosis (the Guvenen higher-order moments the paper names), age profiles, horizon-2 mobility — are silently ungated: declare them reported-not-gated at gate 1 or commit references. 6. The pass rule is exploitable as written. (a) "Every geometry threshold holds on at least 4 of 5 paired seeds": the per-threshold reading passes a candidate that fails a different metric on every seed (0/5 clean seeds). (b) "Pooled holdout" is not an object the harness defines: five independent 20% person-draws overlap (union = 1 - 0.8^5 = 67% of persons, expected multiplicity 1.49) and include persons who were training data for the other seeds' fits. (c) Unstated: that the candidate refits per seed on the 80% complement with the holdout excluded from all fitting and calibration; that split_panel_by_person(fraction=0.2, seed=s) with the picked 20% defines the holdout; that the view filter applies at scoring; and whether the battery compares candidate vs the committed reference (YAML rationale) or vs the holdout battery (pass rule). Concrete amendments before flipping
The two missing floor artifacts (20%-scale window-2, family window-3) take minutes with the committed pipeline; compute-then-lock beats lock-then-amend. |
The review on this pull request found the proposed thresholds did not derive from their stated rationale and were calibrated at the wrong scale (full-panel half-vs-half, ~51k windows per side, while the protocol scores ~20k-window holdouts). This amendment locks compute-then-lock: - Two new committed floor artifacts, built by scripts/build_gate1_floor_artifacts.py and pinned by reproduction tests: the candidate-context window-2 floor (real vs real at ~20%-of-persons scale) and the family window-3 runs floor (81,654 windows). - Tighten c2st_auc_max 0.55 -> 0.53 and prdc_coverage_min 0.90 -> 0.93; keep energy/q99/w1 with rationale re-derived from the ctx20 floor. Every threshold now states its floor mean + k*sd derivation. - Lock geometry on a family window-3 runs view (c2st 0.53, coverage 0.93): window-2 geometry cannot see chained-model persistence understatement. - Battery: add autocorr_log_4yr (0.06) and mean_spell_length (0.25) tolerances; pin the mobility-diagonal definition (zero bin included); annotate zero_persistence = 1 - exit_rate as one constraint; declare unreferenced families reported-not-gated. - Protocol: seed-level conjunction pass rule, named split and scoring calls, candidate refits per seed on the 80% complement, candidate panel pinned to holdout scale; drop the undefined "pooled holdout". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Amendment applied in ec70b0a, item by item against the review table:
Both new artifacts were rebuilt from the committed pipeline ( Awaiting maintainer ratification — merging remains the lock event. |
|
Verdict: AMEND BEFORE RATIFYING. Round-2 review of the amended lock (ec70b0a). Reproduction anchor: all 8 stated 1. Runs-view thresholds are calibrated at 5× deployment scale — round 1's own medicine, reintroduced. 2. Fixing the runs scale destroys the runs view's bite — it cannot do both at 20%. I ran round-1's chained-Markov clone (ρ=0.730, the paper's named failure mode) at protocol scale on both views under the seed-level conjunction. It passes all window-2 geometry (pairs c2st 0.501-0.510) and is caught only by runs c2st 0.533-0.542 > 0.53, on 5/5 seeds. But a runs threshold placed ≥4 deployment-sd above noise is ~0.539-0.540, and the clone (min 0.533) then passes 4/5 seeds. So 3. Conjunction rule is fair to a faithful candidate — confirmed, no change needed. A truth-process redraw (permanent + AR transitory) passes 5/5 under the seed-level conjunction across both views (pairs c2st ≤ 0.511, runs c2st ≤ 0.512, all tolerances clear). Round-1's broken candidates still fail 5/5 under the amended 0.53/0.93. The k<2 tail thresholds do not clip the faithful redraw. The 4. Null-mismatch claim does not hold — reporting the negative. Deployment scores the same persons as the holdout (matching age rows); the floor pairs disjoint persons. I scored one identical donor-bootstrap generator both ways: same-persons c2st 0.670 vs disjoint-persons 0.666 (gap +0.004), energy +0.002, coverage −0.002. Age is too coarse a covariate for the same-persons match to make the comparison materially harder, so the disjoint floor is not anti-conservative for c2st/energy/coverage. The pairs 0.53/0.93 null is defensible. (ctx20 seed-overlap is Jaccard 0.11 / 71% reuse; the disjoint-draw c2st sd is larger at 0.0058, so the pairs c2st lock is +3.7 disjoint-sd — still safe — but the committed energy sd 0.0025 is ~8× the independent-draw sd 0.0003, i.e. inflated by the overlapping construction, not a clean noise estimate. Minor.) 5. Two smaller gaps (fresh eyes). (a) Concrete amendments before flipping
Pairs geometry, battery, conjunction rule, and protocol wording are ready. The runs-view scale is the one blocker: it is the exact defect round 1 caught, and the artifact to fix it takes minutes with the committed pipeline. |
Round 2 (comment 4885102386) found the runs-view thresholds calibrated against the full-panel half-vs-half floor — the same wrong-scale defect round 1 rejected for the pairs view — and showed the runs view cannot be both scale-honest and clone-catching at deployment scale. - Commit a deployment-scale window-3 floor (runs/noise_floor_psid_family_runs_ctx20_9822.json) and re-derive the runs thresholds from it at the same ~4-sd discipline as every other threshold: c2st 0.54 (mean + 4.3 sd), coverage 0.90 (mean - 4.0 sd). - Reattribute the chained-clone guard to the battery, where it belongs: 10yr autocorr gap 4.7x tolerance, 4yr 2.1x, 5/5 seeds; the runs view is labeled a supporting check (the clone straddles its threshold at deployment scale). - Lock q90_ratio_range [0.85, 1.15] on the pairs view (it was computed but ungated). - Make every geometry derivation machine-checkable: a `derivations` block per view in gates.yaml plus tests/test_gates_derivations.py asserting threshold == floor mean + stated k * sd, that every locked threshold has a derivation, that battery tolerances have committed references, and the zero_persistence = 1 - exit_rate identity. - Document the ctx20 overlapping-draw construction (disjoint-draw c2st sd 0.0058 keeps the pairs lock at +3.3 sd; the committed energy sd is ~8x the independent-draw sd, i.e. conservative). - Restate the pairs q99 lower bound as k = -2.56 (the prior -2.6 rounds to 0.69, not the locked 0.70). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Round-2 amendment applied in d507c22, item by item against the review table:
Findings 3 (conjunction fairness) and 4 (null-mismatch negative) required no change; the faithful-redraw and broken-candidate results are now cited in the YAML rationale as review evidence. Full suite: 91 passed (was 79; +11 derivation-binding, +1 runs_ctx20 reproduction pin). Awaiting round-3 referee, then maintainer ratification — merging remains the lock event. |
|
Verdict: RATIFY AS-IS. Non-blocking notes in findings 4 and 6. Round-3 review of amendment d507c22 against round 2 (comment 4885102386). Reproduction anchor: full suite 91 passed locally (79 + 11 derivation-binding + 1 runs_ctx20 pin, as the amendment reply states); CI runs the derivations test on every PR without PSID. 1. Round-2 table lands item-by-item; the diff is exactly scoped. gates.yaml ec70b0a→d507c22 changes only: the runs 2. All ten derivations reproduce from the committed artifacts at exact float precision. Pairs: 0.5108054+4.2·0.0045535=0.52993→0.53; 0.0033793+1.8·0.0025143=0.007905→0.008; 0.9577148−4.7·0.0059362=0.92981→0.93; q99 1.0428391−2.56·0.1337886=0.70034→0.70 (the restatement was forced: −2.6 gives 0.69494→0.69) and +2.7→1.40407→1.40; q90 −3.3→0.84812→0.85 and +3.0→1.15033→1.15; w1 +1.9→0.05973→0.06. Runs: 0.5091816+4.3·0.0072226=0.54024→0.54; 0.9506836−4.0·0.0125394=0.90053→0.90. Every floor stat quoted in the YAML comments and the PR body matches its artifact at the stated precision, and the +3.3 disjoint-sd note is self-consistent ((0.53−0.5108)/0.0058=3.31), conservative vs round 2's +3.7. 3. The divergence from round 2's literal option (a) — re-deriving the runs view at ~4σ instead of confessing +2.9σ — is the right call, and the loosened view keeps bite. Keep it; do not drop it.
4. The derivations test binds, with characterized limits that are acceptable by design. Mutating the runs c2st floor mean by +0.01 (1.4 floor-sd) fails the test. Limits: (a) rounding deadband — a mean drift under half a rounding unit (±0.005 absolute; 0.01–1.1 floor-sd depending on metric) passes silently; any artifact regeneration is itself a public diff, and the PSID-staged reproduction pins catch seed-0 value changes exactly. (b) A post-hoc k edit re-fits a shifted artifact (verified: mean +0.01 with k 4.3→2.88 passes 11/11) — by design: the test enforces threshold↔k↔artifact consistency, while immutability comes from the public-amendment rule in the file header. (c) One uncovered vector: hand-editing an artifact's 5. Clone-guard reattribution verified. battery_reference: ac2yr 0.730124, ac4yr 0.657167, ac10yr 0.538879. Markov-implied clone: 0.730^5 = 0.207, gap 0.332 = 4.7× the 0.07 band; 0.730^2 = 0.533, gap 0.124 = 2.1× the 0.06 band — both as stated. Empirical rerun: ac4yr fails 5/5 (clone 0.536–0.546), ac10yr fails 5/5 (0.203–0.234). No remaining text claims the runs view catches the clone: thresholds block, PR body, and commit message all state straddle-plus-battery. 6. Non-blocking notes. (a) The views comment (gates.yaml:43–49) still motivates the runs lock with "because window-2 geometry cannot see chained-model persistence understatement" — literally true of window-2, but a hostile reader could juxtapose it against the thresholds block's "SUPPORTING check, not the guard"; one clarifying clause closes it, and the dated, review-citing thresholds block controls the numbers either way. (b) Remaining computed-but-undeclared quantities: the runs-view scorecard also emits energy/q99/q90/w1 (present in the runs_ctx20 artifact) with no lock and no reported_not_gated line — both prior rounds prescribed runs = {c2st, coverage}, and two-step marginals follow from correct one-step conditionals, so this is declaration-only; likewise change_moments registers Every locked number derives from committed evidence and is machine-checked; the amendment fixed what round 2 found without disturbing anything round 2 cleared. Ratify by merging; flip |
|
Third-round review (independent machine): ratification-ready.
Nothing further from review. Per the PR's own terms, merging is the lock — that signature belongs to the ratifier, not the reviewer. |
Locks the gate-1 numbers. Do not merge casually: merging this PR IS the threshold lock — the pre-registration event that must precede the first model run.
Amended twice under adversarial review, both rounds applied before ratification: round 1 (thresholds didn't derive from their stated rule; floor at the wrong scale for the pairs view) and round 2 (the same wrong-scale defect reintroduced on the runs view; clone-guard misattribution; q90 ungated; derivations not machine-checked). Every threshold now derives from a committed deployment-scale floor artifact, and
tests/test_gates_derivations.pyasserts each locked value equals its stated floor mean + k·sd — the rationale can no longer drift from the numbers.Geometry,
psid_family_earnings_pairs(window 2) — floor =runs/noise_floor_psid_family_ctx20_9822.json(real vs real at ~20%-of-persons scale):Geometry,
psid_family_earnings_runs(window 3) — floor =runs/noise_floor_psid_family_runs_ctx20_9822.json(deployment scale, per round 2): C2ST ≤ 0.54 (0.5092 + 4.3σ), coverage ≥ 0.90 (0.9507 − 4.0σ). The runs view is a supporting check: at deployment scale a chained-Markov clone straddles its C2ST threshold (0.533–0.542 vs 0.54). The pre-registered guards for chained-model persistence understatement are the battery's autocorrelation and spell tolerances — clone 10yr autocorr 0.207 vs reference 0.539 (4.7× the ±0.07 band) and 4yr 0.533 vs 0.657 (2.1× the ±0.06 band), failing 5/5 seeds.Battery (absolute bands around
battery_referenceinruns/noise_floor_psid_family_9822.json): autocorr 2yr ± 0.05, 4yr ± 0.06, 10yr ± 0.07; mobility diagonal ± 0.05 (six origin-diagonal cells, zero bin included); zero persistence ± 0.05 and exit ± 0.05 (one constraint — the identityzero_persistence = 1 − exit_rateis now test-pinned); entry ± 0.02; mean spell length ± 0.25. Change moments (sd/skew/kurtosis), age profiles, and horizon-2 mobility are reported-not-gated.Pass rule: seed-level conjunction — a seed passes iff every locked threshold on every locked view holds; the gate passes iff ≥4/5 seeds pass geometry and ≥4/5 pass the battery. The candidate refits per seed on the 80% complement and emits trajectories for exactly the holdout persons. Round 2 verified fairness empirically: a faithful truth-process redraw passes 5/5 under the full conjunction; round 1's broken candidates (shuffled dynamics; 18% persistence understatement) fail 5/5.
After merge, flipping
locked: truehappens in a one-line follow-up before any model run.🤖 Generated with Claude Code