Guard the cross-engine spotcheck on the artifact's pe-us pin - #41
Merged
Conversation
The spotcheck regenerates engine PIAs and compares them to the committed artifact, which pins a policyengine-us revision. Against a checkout at any other revision the comparison is meaningless and failed as if the artifact were stale; it now skips with the pinned and found revisions named. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
MaxGhenis
added a commit
that referenced
this pull request
Jul 6, 2026
The referee (PR #57 review) found two text-level defects, no numeric ones. First, the +/-5% bands claimed the paper as their source; the paper carries no such criterion. The a_priori_source now states the exact provenance — the DRAFT Phase-1 acceptance tables, in AIME units — notes that PIA concavity makes a 5% PIA-space band stricter than the draft figure (roughly 7-9% AIME-equivalent), and declares the bands as this amendment's own proposal-time criterion. The source key renames to amendment_5pct_criterion accordingly. Second, the runs-view c2st demotion silently changed the geometry-block verdicts of four committed runs; a demotion_disclosure block now lists every flip, shows that no overall gate-1 verdict changes (each run still fails the battery and, under the amendment, the benefit-space block — referee-verified baseline pooled Q0 +19.0), and acknowledges where the anti-persistence duty concentrates. Also aligns the anchor reproduction test's pe-us pin check with the repo precedent (PR #41): skip with both revisions named on a mismatched checkout instead of failing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Jul 6, 2026
* Gate-1 amendment proposal: decision-relevant recalibration Amendment OBJECT under the locked pre-registration contract. It PROPOSES changes and changes NOTHING: the locked gate_1.thresholds block is untouched (locked: true stays; every locked value byte-identical), no model reads the new subsection, and nothing takes effect until a fresh adversarial referee round runs on this proposal and the maintainer ratifies by merging. New committed anchor (real-vs-real PIA-proxy floor at deployment scale): - runs/pia_proxy_floor_9822.json + scripts/build_pia_proxy_floor.py: the ctx20 construction (40% draw seed 1000+s, halved at seed s) on the FULL locked filtered panel, both disjoint halves pushed through the pinned PIA-proxy functional imported verbatim from build_downstream_relevance (single source of truth). Records per seed and mean/sd: |mean % gap| 1.86+/-0.76, |median % gap| 1.61+/-0.99, weighted KS 0.0288+/-0.0078, and the Q0 (zero-anchor subgroup) gap (pooled |Q0| 2.66%; per-seed 5.25+/-4.53). The floor is real-vs-real, so the oracle import path does NOT pull populace.fit; the reproduction test needs only PSID and runs live (seed 0, pinned to float precision). gates.yaml gate_1.amendment_proposed (pure addition, zero deletions): - change 1: runs-view c2st_auc_max moves gated -> reported-not-gated (PR #56: candidate-7 KS inside the real-real noise floor, central benefit gaps <=1.9%; the window-3 residual is benefit-immaterial texture per PR #54). - change 2: a NEW gated benefit_space block on the candidate PIA-proxy panels: |mean %|<=5, |median %|<=5, deciles d3-d9 <=5 (d1 AND d2 reported-not-gated -- the new anchor shows their real-real floor clips 5% on 5/5 and 3/5 seeds), weighted KS <= committed anchor mean + 4*sd (machine-checkable derivation = 0.0599), and pooled |Q0 mean %|<=5. The 5% bands cite the paper's pre-stated +/-5%-of-SSA-statistics success criterion (docs Phase-1 acceptance table); only KS is floor-derived. Candidate 7 measured +9.3% on Q0, so the amendment ADDS strictness where the evidence says it bites -- recalibration toward decision-relevance, not relaxation. Discriminator check on committed evidence: real-vs-real PASSES the block, candidate 7 FAILS it on Q0. tests/test_gates_derivations.py: binds the proposed block -- the KS band equals anchor mean + stated k*sd, every proposed gated metric carries a derivation or an a-priori-source citation, the anchor's stats recompute from its stored per-seed values, and the reported-not-gated d1/d2 partition matches the anchor's per-decile fragility. ruff + black -l 79 clean; full pytest green in the repo .venv (275 passed, 9 skipped -- the skips are pre-existing populace.fit gate reproductions). Evidence chain: PR #54, PR #55, PR #56, issue #42. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Apply the amendment referee round: honest sourcing and full disclosure The referee (PR #57 review) found two text-level defects, no numeric ones. First, the +/-5% bands claimed the paper as their source; the paper carries no such criterion. The a_priori_source now states the exact provenance — the DRAFT Phase-1 acceptance tables, in AIME units — notes that PIA concavity makes a 5% PIA-space band stricter than the draft figure (roughly 7-9% AIME-equivalent), and declares the bands as this amendment's own proposal-time criterion. The source key renames to amendment_5pct_criterion accordingly. Second, the runs-view c2st demotion silently changed the geometry-block verdicts of four committed runs; a demotion_disclosure block now lists every flip, shows that no overall gate-1 verdict changes (each run still fails the battery and, under the amendment, the benefit-space block — referee-verified baseline pooled Q0 +19.0), and acknowledges where the anti-persistence duty concentrates. Also aligns the anchor reproduction test's pe-us pin check with the repo precedent (PR #41): skip with both revisions named on a mismatched checkout instead of failing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The spotcheck failed spuriously against any policyengine-us checkout other than the artifact's pinned revision (bf71be3b). It now skips with both revisions named unless the pin matches. Verified both ways locally: skips on a stale default checkout, 11/11 pass with POPULACE_DYNAMICS_PE_US_DIR at the pinned revision.
🤖 Generated with Claude Code