UK local candidate gate battery, release diagnostics, and scorer (#147 PR 1) - #830
Conversation
|
Automated review pass (Claude Code, high effort, full non-test diff — no execution). Four findings. Two of them are patterns this lane has already fixed elsewhere, recurring in new code. 1.
|
|
Thanks — worked through all four. 1, 2 and 4 are fixed in f169ddc; 3 I dispute, with evidence and a regression test so the question stays settled either way. 1. Positional join in the scorer — confirmed, fixedCorrect, and correctly prioritised:
The regression uses deliberately asymmetric fixtures so position and join actually disagree — a row-reversed metrics file scores identically through the join, where positional pairing would return 8.0 and 50.0 instead of 10.0 and 70.0. Those joined values are pinned directly, so a silent revert fails rather than drifts. Four refusal tests cover missing 2. Unwired holdout — confirmed, fixedAgreed on both the diagnosis and that this PR is what makes it actionable. The scorer can't run solves itself (it has neither the frame nor the matrix), but it doesn't need to: the driver publishes The receipt now:
3. Diagnostic entries get no status check — I don't think this one holdsThe status = outcome.get("status")
if status not in _UK_GATE_BATTERY_STATUSES: # "error" is not in the taxonomy
failures.append(...); continue
if status == "unreached": failures.append(...)
if status == "not_applicable": failures.append(...)So both statuses the finding names are already refused for The deleted comment is a fair hit, though — it's what made this legible, and its concern (a relabel dodging the shippability recompute) is now handled more tightly than before, since criticality is pinned per entry against the committed spec and a relabel in either direction fails. I've restored documentation saying exactly that, and added two tests so this isn't re-litigated from reading alone: one asserting a diagnostic entry carrying If you meant something narrower — e.g. that a diagnostic gate reporting 4. Fixed key over a parameterised limit — confirmed, fixedReal, and the existing test showed it: a key named Verificationruff clean; battery/spec 398 passed; UK local lane 285 passed, 2 skipped (up 7 — the new regressions); |
|
Verification pass over First: your dispute on finding 3 was right, and I checked it rather than conceding. At Findings 1 and 4 verify.
The holdout is a real held-out quantity, which was the thing worth checking: Three findings, all in the new scorer code. 1.
|
f169ddc to
e212e37
Compare
|
All three fixed in 5514f41c, and the branch is rebased onto main now that #827 has landed ( 1 + 2 — same root cause, fixed at the rootThese are one bug wearing two hats, and the fix is the same for both: the scorer was carrying its own copy of the objective. It had a private One correction to the diagnosis, and it makes the finding more worth fixing rather than less: the two are not on different scales today — they are the same formula to the float. So a reader differencing them today gets a meaningful number by luck. What makes it a real defect is the second half of your point: they are independent constants, and So rather than renaming the label (finding 2's suggested fix), which would have preserved the duplication and just described it more carefully, the scorer now calls
3 — confirmed, and it caught a real inconsistency immediatelyRight, and right that it's the merge-blocker: the refusal guaranteed presence, not consistency, so a fitted number with plausible folds beside it satisfied everything. Worth reporting: the check immediately failed my own fixture, which declared RebaseRebased onto Verification (post-rebase)battery/spec 398 passed; UK local + cross-grain lane 317 passed, 2 skipped; |
|
Verification pass over Both fixes hold, and the root-cause framing was the better call. Deleting the private cap and calling The consistency check holds where it matters. Two findings. 1.
|
…l scorer (#147) The rowwise candidate driver's hardcoded gate verdicts become a real scoped battery: UK_LOCAL_GATE_SCOPE declares six terminal entries (ladder post-calibration and both-grain area support release-blocking; target fit, per-family fit, weight ratio, and weight ESS diagnostic until the first #762 measured-fit receipt arms them), evaluated through the shared executor and written as a signed local_candidate report. area_support is the one new gate function; the local scope is classified out of the national certification until #146. Candidate bundles gain the schema-v6 calibration diagnostics payload, weakest-family and bottom-15 weakest-area rollups with country legs, and the report-only rotated 5-fold holdout; score_uk_local_candidate scores candidates against the frozen 17,077-cell local register. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…asured holdout (#147) The local scorer paired the incumbent wide weights with the household metric table by row position behind a row-count check, which passes for any permutation of the right size — the uk-data#468 shape, and worst in a scorer, where a mis-joined incumbent reads as a candidate that won or lost rather than as an error. Both tables now carry household_id and are joined on it, with the join required to be unique and total on both sides. The score receipt reported one loss under two names with holdout_basis "none_declared" while this change adds the rotation. The scorer now requires the rotated holdout the candidate driver already publishes in the same schema-v6 payload, reports its measured mean and basis, and labels the head-to-head counters as the candidate's fitted surface — the incumbent is never re-solved, so it has no holdout and says so. Also: key the weakest-area list by role and carry its own limit, so the name cannot assert a count the list does not have; and restore the documentation of what the diagnostic criticality label does and does not exempt, with tests pinning that unreached, not_applicable, and out-of-taxonomy statuses are refused for diagnostic entries too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se (#147) The scorer carried its own copy of the objective — a private UK_LOCAL_SCORE_LOSS_CAP and a hand-rolled capped mean absolute relative error — beside a holdout computed by the canonical relative_error_loss at the doctrine cap. The two agree today, but they are independent constants, and #762 is chartered to adjudicate the doctrine cap, at which point the receipt would have reported two numbers on different scales under names inviting a difference. The scorer now calls relative_error_loss at UK_LOCAL_TARGET_LOSS_CAP for both aggregates, so the objective label names the function that actually produced every loss in the receipt, and the private cap is gone rather than kept in sync by hand. The holdout block was checked for presence but not for consistency: fold losses were length-checked only, and the headline mean and worst were never reconciled against them, so a fitted number sitting above plausible folds satisfied every check. Folds must now be finite and non-negative, the mean must close over them, and the worst must be their maximum — the invariants summarize_rotations establishes when it derives the summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… scale (#147) The scorer's objective label vouched for every loss in the receipt including the holdout, but the holdout is a number recorded upstream under whatever cap that run used, and nothing checked it. The same scale disagreement the last round closed inside the module therefore survived across the module boundary, scheduled to appear the moment #762 moves the doctrine cap: the fitted-surface aggregates would follow the constant and a pinned diagnostics file would not. rotated_uk_local_holdout now declares the cap its folds were measured under, and the scorer refuses a holdout whose declared cap is not the cap it reports its own aggregates at — re-measure the candidate rather than reporting the two on different scales. The per-target drift rows kept a hand-rolled max(abs(target), 1) denominator, which is a second copy of the canonical scale even though the quantity around it is legitimately different (signed and uncapped). It now imports default_target_loss_scales, so the drift rows and win counters cannot drift away from the aggregates printed beside them, and the docstring's claim is literally true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e212e37 to
8a2f9b3
Compare
|
Both fixed in 8a2f9b3, and the branch is rebased onto 1 — the holdout's cap, now carried and enforcedConfirmed, and checking the producer turned up something your diff-only read couldn't see: Fixed on both sides, taking your second option:
Your framing of why the second option is better was the deciding argument — it makes the cross-module agreement an enforced invariant rather than a convention, the same move as requiring the rotation rather than declaring it away. And the failure mode it produces is the right one: when #762 moves the cap, a stale rotation stops the scorer instead of quietly being reported beside freshly-computed aggregates. The candidate needs re-measuring at that point, which is exactly what the refusal says. The declared cap is also echoed into the receipt's 2 — the row scale, imported rather than restatedRight, and cheap to close: Noted on Rebase onto #829Six conflicts, one of which deserves flagging:
Verification (post-rebase)battery/spec 404 passed; UK local + cross-grain + UC-capital lane 329 passed, 2 skipped; |
First of two PRs for #147 (plan:
repos/uk-147-local-credibility-plan.md, approved 2026-08-31) — the gate tooling half. The target-activation half (parity register, masking doctrine, council-tax/PIPR surface work) follows separately, stacked on the #802 branch.What this does
UK_LOCAL_GATE_SCOPE— a fourth scoped battery beside spine/calibration/national, evaluated atterminalbytools/build_uk_rowwise_candidate.pyand written as a signedlocal_candidatereport (<candidate>.local_gates.json). The driver's previous hardcoded"passed"receipt-pointer verdicts are gone; Logbookgate_verdictsnow come from battery statuses, with scope-coverage refusals. Write-then-block held: on a blocked phase the finalized report is still persisted before the raise.uk_local_geography_ladder_post_calibration(release-blocking; existing ladder evaluator adapted viaUKGateBinding, legacy nameuk_geography_ladder),uk_local_area_support(release-blocking at both grains: ESS ≥ 50, rows ≥ 50, distinct sources ≥ 50 — María's D4/D5 ruling; constituency K=4 measured minima were 101/86.4/97 per the K=1 rowwise pilot machinery: clone the spine, assign the ladder, measure per-area support (#761) #815 receipts; an LA-grain miss re-opens K with the measurement as justification), and four diagnostic fit gates (target_fit0.25,per_family_fit0.1/0.25,weight_ratio100.0,weight_ess0.01) to be armed by one follow-up citing First calibrated rowwise UK local candidate (#495 increment 6) #762's measured fit.area_supportis the only new gate function: exact-roster fail-closed (a weak area cannot vanish from the evidence), duplicate/invalid refusals, per-area shortfall enumeration._verify_unionstill refuses gaps.calibration_diagnostics.json+uk_diagnosticsblock (geography-level pass rates light upconstituency/local_authority), a weakest-family rollup ranked by loss contribution, a weakest-area-by-fit table (keyed by role, carrying its ownlimit) joined with support/ESS plus per-country legs, andtools/score_uk_local_candidate.pyscoring against the frozen 17,077-cell local register — joining the incumbent onhousehold_id, and requiring and reporting the candidate's measured rotated holdout rather than declaring none.rotated_folds, seed 20260529) with real per-fold training solves — the first caller ofbuild/holdout.py.docs/gate-battery-contract.mdde-staled (13 → 43 entries, four scopes documented); contract pins/digests, entry-gate mirror, diagnostic-id and certification-exclusion sets updated inmicrocosm-data.Verification
ruff check .clean; changed files pass the scoped format check (the repo-wideruff format --checkdebt of ~126 files predates this branch — verified red on clean main).-k "uk_local or uk_terminal or uk_release_certification or diagnostics or uk_rowwise"278 passed, 2 skipped; new/rewired test files 33 passed; fullmicrocosm-datacontract suite 100%;tools/ci_test_groups.py --verifyok (new tests land in the UK lane).--dry-runneeds the staging H5 (not present in this worktree); the driver rewire is covered by synthetic integration tests instead.Review notes
spine_agreement(bindings adapt, never re-implement; result re-mints underuk_geography_ladder) — flagging the name choice for review.household_idrather than paired by row position (the uk-data#468 shape), the measured holdout is wired into the score receipt instead ofnone_declared, and the weakest-area key no longer hardcodes a count. The fourth finding (diagnostic entries going status-unchecked) did not reproduce:unreached/not_applicable/out-of-taxonomy are refused above the criticality branch for every entry; documentation and two regression tests were added rather than a behaviour change.uk_local_weight_ratiouses the national gate'smaximum_max_to_median_ratiovocabulary at the doctrine constant 100.0; max/median is not the doctrine's solver stretch quantity. It ships diagnostic — the First calibrated rowwise UK local candidate (#495 increment 6) #762 arming follow-up should decide whether it should instead read the diagnostics'realized_max_weight_ratio.Refs #147, #495, #665. Not for merge without María's explicit go.
🤖 Generated with Claude Code