Skip to content

UK local candidate gate battery, release diagnostics, and scorer (#147 PR 1) - #830

Merged
juaristi22 merged 4 commits into
mainfrom
uk-local-gate-battery-147
Sep 1, 2026
Merged

UK local candidate gate battery, release diagnostics, and scorer (#147 PR 1)#830
juaristi22 merged 4 commits into
mainfrom
uk-local-gate-battery-147

Conversation

@juaristi22

@juaristi22 juaristi22 commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

First of two PRs for #147 (plan: repos/uk-147-local-credibility-plan.md, approved 2026-08-31) — the gate tooling half. The target-activation half (parity register, masking doctrine, council-tax/PIPR surface work) follows separately, stacked on the #802 branch.

What this does

  • UK_LOCAL_GATE_SCOPE — a fourth scoped battery beside spine/calibration/national, evaluated at terminal by tools/build_uk_rowwise_candidate.py and written as a signed local_candidate report (<candidate>.local_gates.json). The driver's previous hardcoded "passed" receipt-pointer verdicts are gone; Logbook gate_verdicts now come from battery statuses, with scope-coverage refusals. Write-then-block held: on a blocked phase the finalized report is still persisted before the raise.
  • Six declared entries per the pre-ruled A2 sheet: uk_local_geography_ladder_post_calibration (release-blocking; existing ladder evaluator adapted via UKGateBinding, legacy name uk_geography_ladder), uk_local_area_support (release-blocking at both grains: ESS ≥ 50, rows ≥ 50, distinct sources ≥ 50 — María's D4/D5 ruling; constituency K=4 measured minima were 101/86.4/97 per the K=1 rowwise pilot machinery: clone the spine, assign the ladder, measure per-area support (#761) #815 receipts; an LA-grain miss re-opens K with the measurement as justification), and four diagnostic fit gates (target_fit 0.25, per_family_fit 0.1/0.25, weight_ratio 100.0, weight_ess 0.01) to be armed by one follow-up citing First calibrated rowwise UK local candidate (#495 increment 6) #762's measured fit.
  • area_support is the only new gate function: exact-roster fail-closed (a weak area cannot vanish from the evidence), duplicate/invalid refusals, per-area shortfall enumeration.
  • Certification accounting: the local scope is explicitly classified out of the national release certification until Publish a Populace UK local variant with rowwise H5 and long weights #146 — closed-world in both directions, so _verify_union still refuses gaps.
  • I4 diagnostics: candidate bundles now emit the shared schema-v6 calibration_diagnostics.json + uk_diagnostics block (geography-level pass rates light up constituency/local_authority), a weakest-family rollup ranked by loss contribution, a weakest-area-by-fit table (keyed by role, carrying its own limit) joined with support/ESS plus per-country legs, and tools/score_uk_local_candidate.py scoring against the frozen 17,077-cell local register — joining the incumbent on household_id, and requiring and reporting the candidate's measured rotated holdout rather than declaring none.
  • D10 rider: report-only rotated 5-fold holdout (rotated_folds, seed 20260529) with real per-fold training solves — the first caller of build/holdout.py.
  • Riders: docs/gate-battery-contract.md de-staled (13 → 43 entries, four scopes documented); contract pins/digests, entry-gate mirror, diagnostic-id and certification-exclusion sets updated in microcosm-data.

Verification

  • ruff check . clean; changed files pass the scoped format check (the repo-wide ruff format --check debt of ~126 files predates this branch — verified red on clean main).
  • Battery/spec suite 398 passed; -k "uk_local or uk_terminal or uk_release_certification or diagnostics or uk_rowwise" 278 passed, 2 skipped; new/rewired test files 33 passed; full microcosm-data contract suite 100%; tools/ci_test_groups.py --verify ok (new tests land in the UK lane).
  • The end-to-end --dry-run needs the staging H5 (not present in this worktree); the driver rewire is covered by synthetic integration tests instead.

Review notes

  • The ladder entry reuses the allowed gate name spine_agreement (bindings adapt, never re-implement; result re-mints under uk_geography_ladder) — flagging the name choice for review.
  • Update (f169ddc): Vahid's review round is addressed — the incumbent is now joined on household_id rather than paired by row position (the uk-data#468 shape), the measured holdout is wired into the score receipt instead of none_declared, and the weakest-area key no longer hardcodes a count. The fourth finding (diagnostic entries going status-unchecked) did not reproduce: unreached/not_applicable/out-of-taxonomy are refused above the criticality branch for every entry; documentation and two regression tests were added rather than a behaviour change.
  • uk_local_weight_ratio uses the national gate's maximum_max_to_median_ratio vocabulary at the doctrine constant 100.0; max/median is not the doctrine's solver stretch quantity. It ships diagnostic — the First calibrated rowwise UK local candidate (#495 increment 6) #762 arming follow-up should decide whether it should instead read the diagnostics' realized_max_weight_ratio.
  • Implemented by Codex under the reviewed plan's protocol; every deviation was verified (sandbox uv-cache path; no UK golden spec exists — pins live in the spec/contract tests; pre-existing format debt untouched).

Refs #147, #495, #665. Not for merge without María's explicit go.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Automated review pass (Claude Code, high effort, full non-test diff — no execution). Four findings. Two of them are patterns this lane has already fixed elsewhere, recurring in new code.

1. tools/score_uk_local_candidate.py:112 — the incumbent is scored on a positional join

_incumbent_estimates pairs the incumbent wide-weight table with the household metric table by row position: len(weights) != len(metrics) is the only check, and then np.dot(weights[area], metrics[measure]) runs. Neither CSV is joined on household_id.

So any row-order difference between the two files — or between either and the candidate frame — silently produces a well-formed-looking incumbent score computed from mismatched households. That is the policyengine-uk-data#468 failure exactly, in local-area code, and #468 is still open and unfixed upstream. The scorer is also the worst place for it: a mis-joined incumbent does not look broken, it looks like a candidate that won or lost, and the rescale-style national totals that usually mask this kind of thing are not even in play.

Joining both tables on household_id and asserting the join is total is the fix; the length check that is there now passes for any permutation of the right size.

2. tools/score_uk_local_candidate.py:250 — the holdout is unwired, and this PR adds the thing it needs

The score receipt reports candidate_train_loss and candidate_full_loss as the same variable, candidate_holdout_loss: None, and holdout_basis: "none_declared". So the candidate is compared to the incumbent only on the surface it was fitted to, and the win counters favour it by construction.

This is the same shape as the #743 rule-1 scorer finding, with one difference that makes it more actionable: this PR adds rotated_uk_local_holdout in the same change, so a genuine held-out loss now exists in the codebase and simply is not connected to the scorer. Wiring it in, or refusing to score when holdout_basis == "none_declared", both resolve it — but shipping a scorer with an unwired holdout beside a newly-added holdout rotation seems like the one option that leaves the receipt claiming more than it measured.

3. packages/microcosm-data/src/microcosm/data/contract.py:2584 — the four diagnostic gates get no status check at all

The new expected_criticality branch checks the label, then only recomputes shippability when criticality == "release_blocking". For the four _UK_GATE_BATTERY_DIAGNOSTIC_IDS entries, no status check runs on any path — and the comment that previously documented not_applicable being enforced globally was deleted in the same hunk.

The consequence is that uk_local_target_fit and its three siblings pass contract validation carrying status error or not_applicable, i.e. having compared nothing. That is the green-by-absence class arriving through a refactor rather than through a missing threshold, and the deleted comment is what would otherwise have flagged it in review.

4. packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py:186 — a fixed key name over a parameterised limit

uk_weakest_areas_by_fit returns the key "bottom_15" while limit is a parameter. A caller passing any other limit publishes a differently-sized list under a name asserting 15, so neither a reader nor a downstream pin can tell how many areas were actually reported. Deriving the key from limit, or recording limit alongside the list, keeps the two from disagreeing.


1 and 2 I would resolve before merge. 1 because a positional join in a scorer produces plausible wrong answers rather than errors, and 2 because the receipt names a comparison it did not make while the ingredient for making it landed in the same PR.

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks — worked through all four. 1, 2 and 4 are fixed in f169ddc; 3 I dispute, with evidence and a regression test so the question stays settled either way.

1. Positional join in the scorer — confirmed, fixed

Correct, and correctly prioritised: len(weights) != len(metrics) passes for every permutation of the right size, so the incumbent was scored from whatever household happened to sit at each row index.

_align_on_household_id now joins both tables on household_id and treats the join as the check: the column is required on both sides (its absence is refused with "positional pairing is not a join"), nulls and duplicate ids are refused, the two household sets must match exactly, and metrics are reindexed onto the weights' order before any dot product runs.

The regression uses deliberately asymmetric fixtures so position and join actually disagree — a row-reversed metrics file scores identically through the join, where positional pairing would return 8.0 and 50.0 instead of 10.0 and 70.0. Those joined values are pinned directly, so a silent revert fails rather than drifts. Four refusal tests cover missing household_id on either side, a partial join, and duplicate ids.

2. Unwired holdout — confirmed, fixed

Agreed on both the diagnosis and that this PR is what makes it actionable. The scorer can't run solves itself (it has neither the frame nor the matrix), but it doesn't need to: the driver publishes uk_diagnostics.rotated_holdout inside the same schema-v6 payload the scorer already reads. So it's wired rather than declared away.

The receipt now:

  • requires the rotation — a candidate whose diagnostics carry no holdout is refused, not scored on its fitted surface alone;
  • reports candidate_holdout_loss and a real holdout_basis (rotated_folds:n_folds=5:seed=20260529), with the full block (folds, seed, per-fold losses, worst) beside it;
  • renames the duplicated candidate_train_loss/candidate_full_loss pair to candidate_fitted_surface_loss, since reporting one number under two names implied a distinction that was never drawn;
  • keeps incumbent_holdout_loss: null with an explicit incumbent_holdout_basis: "none_available_incumbent_not_resolved" — the incumbent is scored from published weights and never re-solved, so it has no holdout, and the receipt says that rather than leaving a suggestive null;
  • labels the win counters head_to_head_surface: "candidate_fitted_surface", so the head-to-head can't be read as the held-out comparison it isn't.

3. Diagnostic entries get no status check — I don't think this one holds

The unreached, not_applicable, and outside-the-taxonomy checks all sit above the criticality branch and run for every entry, diagnostics included:

status = outcome.get("status")
if status not in _UK_GATE_BATTERY_STATUSES:   # "error" is not in the taxonomy
    failures.append(...); continue
if status == "unreached":     failures.append(...)
if status == "not_applicable": failures.append(...)

So both statuses the finding names are already refused for uk_local_target_fit: not_applicable by the third check, error by the first (the taxonomy is passed / failed / not_applicable / evidence_absent / unreached — there is no error status). What the diagnostic label actually buys is narrower than "no status check": only that failed and evidence_absent don't block, which is the declared posture for the four fit gates until #762 arms them.

The deleted comment is a fair hit, though — it's what made this legible, and its concern (a relabel dodging the shippability recompute) is now handled more tightly than before, since criticality is pinned per entry against the committed spec and a relabel in either direction fails. I've restored documentation saying exactly that, and added two tests so this isn't re-litigated from reading alone: one asserting a diagnostic entry carrying not_applicable, unreached, or error is refused, one asserting uk_local_area_support can't be relabelled diagnostic.

If you meant something narrower — e.g. that a diagnostic gate reporting evidence_absent should itself be refused rather than merely non-blocking — say so and I'll take it as a separate question; that's a posture change for the four gates, and #762's arming pass is where it belongs.

4. Fixed key over a parameterised limit — confirmed, fixed

Real, and the existing test showed it: a key named bottom_15 holding 4 rows. The payload is now keyed by role — bottom_by_fit — and carries limit and n_areas_scored alongside, so it can't assert a count it doesn't have. A test passes limit=1 and checks the reported limit follows the list.

Verification

ruff clean; battery/spec 398 passed; UK local lane 285 passed, 2 skipped (up 7 — the new regressions); microcosm-data contract 228 passed (up 2); ci_test_groups.py --verify ok.

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Verification pass over f169ddc4 (Claude Code, high effort; ignoring the merge conflict, reviewing the code).

First: your dispute on finding 3 was right, and I checked it rather than conceding. At f169ddc4, status not in _UK_GATE_BATTERY_STATUSES refuses error as outside the taxonomy, and the unreached and not_applicable refusals both sit above the criticality branch and run for every entry, diagnostics included. The diagnostic label only buys that failed / evidence_absent do not block, which is the declared posture until #762 arms them. My finding was wrong, and the regression test means the question stays settled.

Findings 1 and 4 verify.

  • _align_on_household_id is genuinely total in both directions: the column is required on both sides, nulls refused, uniqueness checked per side via set_index + is_unique, exact set equality asserted, then metrics.reindex(order). The reindex happens once at the top of _incumbent_estimates before the per-spec loop, so every np.dot uses joined order, and pd.read_csv output goes straight into the join — no positional path survives in the scorer. The asymmetric fixture is the right call: pinning 10.0 and 70.0 where positional pairing would give 8.0 and 50.0 means a silent revert fails rather than drifts.
  • uk_weakest_areas_by_fit now returns the role-named bottom_by_fit with limit and n_areas_scored alongside.

The holdout is a real held-out quantity, which was the thing worth checking: rotated_uk_local_holdout re-solves on each fold's training subset via solve_uk_rowwise_weights_under_doctrine and evaluates problem.matrix[holdout_indices] @ train_solve.weights against held targets. Not a re-labelled fitted number. The refusal in _candidate_holdout is a hard raise with no skip branch, and it is reachable.

Three findings, all in the new scorer code.

1. tools/score_uk_local_candidate.py:398 and ~365 — the two losses are on different scales

candidate_holdout_loss and candidate_fitted_surface_loss are reported side by side but computed by different functions: the fitted-surface loss is the scorer's own mean(min(|rel_err|, 10)), while the holdout mean comes from the solver's relative_error_loss(..., target_loss_cap=UK_LOCAL_TARGET_LOSS_CAP). Differencing them — which is the obvious thing a reader or a downstream promote gate will do — yields a meaningless number, and the two caps are independent constants that can drift apart later.

Given the renaming work in this same commit was about not implying distinctions that were not drawn, this is the mirror case: two numbers presented as comparable that are not.

2. tools/score_uk_local_candidate.py:~370 — the objective label names a different function than the value

"objective": "relative_error_loss" labels the scorer's capped mean absolute relative error, which is not the solver's relative_error_loss. That is provenance taken from a different member than the value — the class that produced the latest_plateau metadata bug on #735. A distinct name for the scorer's own objective keeps the two legible.

3. tools/score_uk_local_candidate.py:_candidate_holdout — the folds are not cross-checked against the summary

fold_losses is validated for length only (len(fold_losses) != n_folds). Element values are never checked finite or non-negative, and mean_holdout_loss / worst_holdout_loss are never reconciled against the folds.

So the per-fold numbers are not independently verified, and the substitution the refusal exists to catch still passes: a payload whose mean_holdout_loss is actually a fitted value, with plausible folds sitting beside it, satisfies every check. Asserting mean ≈ fsum(folds)/n and worst == max(folds) closes it, and is the same closure-against-the-parts property that #795's bucket assertion ended up needing.


3 is the one I would fix before merge: the refusal now guarantees a holdout block is present, and this is what would make it guarantee the block is consistent. 1 and 2 are labelling, but both are the kind that mislead a later reader or a gate rather than a person reading the code today.

@juaristi22
juaristi22 force-pushed the uk-local-gate-battery-147 branch from f169ddc to e212e37 Compare September 1, 2026 10:39
@juaristi22

Copy link
Copy Markdown
Collaborator Author

All three fixed in 5514f41c, and the branch is rebased onto main now that #827 has landed (e212e37b). Thanks for re-checking finding 3 rather than taking my word for it — that's the right instinct, and it's why the regression test is there.

1 + 2 — same root cause, fixed at the root

These are one bug wearing two hats, and the fix is the same for both: the scorer was carrying its own copy of the objective. It had a private UK_LOCAL_SCORE_LOSS_CAP = 10.0 and a hand-rolled mean(min(|rel_err|, cap)), sitting beside a holdout computed by the canonical relative_error_loss at UK_LOCAL_TARGET_LOSS_CAP.

One correction to the diagnosis, and it makes the finding more worth fixing rather than less: the two are not on different scales today — they are the same formula to the float. relative_error_loss with default scales is mean(min(|est - tgt| / max(|tgt|, 1), cap)), which is exactly what the scorer reimplemented, and both caps are 10.0. I checked rather than assumed:

canonical 0.05  scorer-reimpl 0.05  identical True

So a reader differencing them today gets a meaningful number by luck. What makes it a real defect is the second half of your point: they are independent constants, and local_doctrine.py says in its own docstring that "the first calibrated rowwise candidate review (#495 increment 6) adjudicates the cap against measured fit" — i.e. #762 is chartered to move UK_LOCAL_TARGET_LOSS_CAP, at which point the scorer's private copy silently does not follow and the receipt starts reporting two scales under names inviting a difference. The luck expires on a scheduled date.

So rather than renaming the label (finding 2's suggested fix), which would have preserved the duplication and just described it more carefully, the scorer now calls relative_error_loss at UK_LOCAL_TARGET_LOSS_CAP for both aggregates, and the private cap is deleted. That resolves 2 as a consequence: "objective" now reads microcosm.calibrate.relative_error_loss and names the function that actually produced every loss in the receipt, including the holdout. It also restores a stated invariant — that function's docstring says it is "the single canonical definition every measurement imports — the solver's closing loss, the acceptance gates, and scorers all call this function." This one didn't.

_relative_error survives for the signed, uncapped per-target drift rows and the head-to-head comparison, which are genuinely a different quantity from the aggregate; its docstring now says so.

3 — confirmed, and it caught a real inconsistency immediately

Right, and right that it's the merge-blocker: the refusal guaranteed presence, not consistency, so a fitted number with plausible folds beside it satisfied everything. _candidate_holdout now requires each fold loss to be finite and non-negative, the mean to close over the folds (math.isclose(mean, fsum(folds)/n)), and the worst to be their maximum — the invariants summarize_rotations establishes when it derives the summary from exactly those folds, so this checks the producer's real property rather than an invented one.

Worth reporting: the check immediately failed my own fixture, which declared worst_holdout_loss: 0.9 over folds topping out at 0.6. That was a fabricated summary sitting in a test, exactly the shape you predicted, and nothing before this caught it. Fixture corrected; four new tests cover a substituted mean, a wrong worst, non-finite/negative folds, and a real summarize_rotations output passing unchanged.

Rebase

Rebased onto da32685c. One real conflict, in tools/build_uk_rowwise_candidate.py: #827 and this PR each added a keyword to _manifest and its call site — resolved by keeping both (cross_grain and calibration_diagnostics). No other overlap; the local gate scope and #827's reconciliation sit side by side in the driver.

Verification (post-rebase)

battery/spec 398 passed; UK local + cross-grain lane 317 passed, 2 skipped; microcosm-data contract 228 passed; ci_test_groups.py --verify ok; ruff clean. The scorer file passes ruff format --check; build_uk_rowwise_candidate.py does not, but it does not on origin/main either — that's part of the repo's ~126-file pre-existing format debt, so I left it rather than mixing a reformat into this diff.

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Verification pass over 5514f41c / e212e37b (Claude Code, high effort; diff-only, no execution).

Both fixes hold, and the root-cause framing was the better call. Deleting the private cap and calling relative_error_loss beats the label rename I suggested, which would have preserved the duplication and just described it more precisely. Verified: no UK_LOCAL_SCORE_LOSS_CAP, no residual 10.0 literal or local stand-in; both fitted-surface aggregates call relative_error_loss(..., target_loss_cap=UK_LOCAL_TARGET_LOSS_CAP); _relative_error is signed and uncapped and feeds only target_drift rows and the win counters, never an aggregate. Your correction on the diagnosis was right too — they were the same formula to the float, and "the luck expires on a scheduled date" is the accurate framing.

The consistency check holds where it matters. _candidate_holdout runs before any estimate or aggregate work, so no path reaches the aggregates without it; n_folds >= 2 plus len(fold_losses) != n_folds rules out an empty list and a count mismatch; rel_tol=1e-9 is tight enough that a substituted mean cannot pass — a swapped-in fitted loss would need nine-digit agreement — and because the mean check jointly pins the fold set, a worst that is the max of different folds cannot slip through either. Choosing the invariants summarize_rotations actually establishes, rather than inventing plausible ones, is why it caught the fabricated fixture.

Two findings.

1. tools/score_uk_local_candidate.py:~430 — the holdout is outside the cap the label claims for it

"objective": "microcosm.calibrate.relative_error_loss" and its comment claim to name the function that produced every loss "including the candidate's holdout". But candidate_holdout_loss is copied verbatim from uk_diagnostics.rotated_holdout.mean_holdout_loss, computed upstream by the candidate driver under whatever cap that run used, and nothing here verifies it.

So the defect you just closed survives across the module boundary: when #762 moves UK_LOCAL_TARGET_LOSS_CAP, the two fitted-surface losses move and the holdout — a recorded number in a pinned diagnostics file — does not. The receipt then carries fitted and held-out figures on different scales while "target_loss_cap" asserts a single one, which is precisely the difference-two-numbers trap, now with the label vouching for it.

Two ways out, either fine: scope the label and target_loss_cap to the aggregates they actually cover and carry the holdout's own declared cap through from the diagnostics; or refuse when the diagnostics do not declare a cap equal to UK_LOCAL_TARGET_LOSS_CAP. The second is more in the spirit of the round — it makes the cross-module agreement an enforced invariant rather than a convention, which is the same move as requiring the rotation rather than declaring it away.

2. tools/score_uk_local_candidate.py:~300 (_relative_error) — a second hand-rolled copy remains, one level down

The docstring says aggregates go through relative_error_loss "so the scorer never carries a second copy of the objective", but max(abs(target), 1.0) is still a hand-rolled copy of that function's default denominator scale. It is a genuinely different quantity — signed, uncapped, as the docstring now says — but not a different scale, so if the canonical scale convention is ever parameterised the drift rows and win counters silently diverge from the aggregates printed beside them.

Same independent-constant shape as the cap, smaller blast radius, and worth closing while the reasoning is fresh: importing the scale rather than restating it would make the docstring's claim literally true.


Not worth changing: math.isclose on worst_holdout_loss against max(folds) is looser than the exact equality the invariant admits, since worst is selected from the folds and so is bit-identical — but at rel_tol=1e-9 nothing wrong can pass.

1 is the one I would fix before merge, for the same reason 1+2 were worth fixing at the root: it is a scale disagreement scheduled to appear rather than a hypothetical one.

juaristi22 and others added 4 commits September 1, 2026 13:50
…l scorer (#147)

The rowwise candidate driver's hardcoded gate verdicts become a real scoped
battery: UK_LOCAL_GATE_SCOPE declares six terminal entries (ladder
post-calibration and both-grain area support release-blocking; target fit,
per-family fit, weight ratio, and weight ESS diagnostic until the first #762
measured-fit receipt arms them), evaluated through the shared executor and
written as a signed local_candidate report. area_support is the one new gate
function; the local scope is classified out of the national certification
until #146. Candidate bundles gain the schema-v6 calibration diagnostics
payload, weakest-family and bottom-15 weakest-area rollups with country legs,
and the report-only rotated 5-fold holdout; score_uk_local_candidate scores
candidates against the frozen 17,077-cell local register.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…asured holdout (#147)

The local scorer paired the incumbent wide weights with the household metric
table by row position behind a row-count check, which passes for any
permutation of the right size — the uk-data#468 shape, and worst in a scorer,
where a mis-joined incumbent reads as a candidate that won or lost rather
than as an error. Both tables now carry household_id and are joined on it,
with the join required to be unique and total on both sides.

The score receipt reported one loss under two names with
holdout_basis "none_declared" while this change adds the rotation. The
scorer now requires the rotated holdout the candidate driver already
publishes in the same schema-v6 payload, reports its measured mean and
basis, and labels the head-to-head counters as the candidate's fitted
surface — the incumbent is never re-solved, so it has no holdout and says so.

Also: key the weakest-area list by role and carry its own limit, so the name
cannot assert a count the list does not have; and restore the documentation
of what the diagnostic criticality label does and does not exempt, with
tests pinning that unreached, not_applicable, and out-of-taxonomy statuses
are refused for diagnostic entries too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se (#147)

The scorer carried its own copy of the objective — a private
UK_LOCAL_SCORE_LOSS_CAP and a hand-rolled capped mean absolute relative
error — beside a holdout computed by the canonical relative_error_loss at
the doctrine cap. The two agree today, but they are independent constants,
and #762 is chartered to adjudicate the doctrine cap, at which point the
receipt would have reported two numbers on different scales under names
inviting a difference. The scorer now calls relative_error_loss at
UK_LOCAL_TARGET_LOSS_CAP for both aggregates, so the objective label names
the function that actually produced every loss in the receipt, and the
private cap is gone rather than kept in sync by hand.

The holdout block was checked for presence but not for consistency:
fold losses were length-checked only, and the headline mean and worst were
never reconciled against them, so a fitted number sitting above plausible
folds satisfied every check. Folds must now be finite and non-negative, the
mean must close over them, and the worst must be their maximum — the
invariants summarize_rotations establishes when it derives the summary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… scale (#147)

The scorer's objective label vouched for every loss in the receipt including
the holdout, but the holdout is a number recorded upstream under whatever cap
that run used, and nothing checked it. The same scale disagreement the last
round closed inside the module therefore survived across the module boundary,
scheduled to appear the moment #762 moves the doctrine cap: the fitted-surface
aggregates would follow the constant and a pinned diagnostics file would not.
rotated_uk_local_holdout now declares the cap its folds were measured under,
and the scorer refuses a holdout whose declared cap is not the cap it reports
its own aggregates at — re-measure the candidate rather than reporting the two
on different scales.

The per-target drift rows kept a hand-rolled max(abs(target), 1) denominator,
which is a second copy of the canonical scale even though the quantity around
it is legitimately different (signed and uncapped). It now imports
default_target_loss_scales, so the drift rows and win counters cannot drift
away from the aggregates printed beside them, and the docstring's claim is
literally true.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-local-gate-battery-147 branch from e212e37 to 8a2f9b3 Compare September 1, 2026 11:59
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Both fixed in 8a2f9b3, and the branch is rebased onto bc43f2f0 now that #829 has landed.

1 — the holdout's cap, now carried and enforced

Confirmed, and checking the producer turned up something your diff-only read couldn't see: rotated_uk_local_holdout was not publishing its cap at all. So there was nothing for the scorer to check even if it had wanted to — the agreement wasn't merely unverified, it was unexpressible.

Fixed on both sides, taking your second option:

  • Producer — the rotation now declares target_loss_cap: UK_LOCAL_TARGET_LOSS_CAP in its block, so the cap its folds were measured under travels with them.
  • Consumer_candidate_holdout requires that declaration and refuses when it is not the cap the scorer reports its own aggregates at: "re-measure the candidate rather than reporting the two on different scales."

Your framing of why the second option is better was the deciding argument — it makes the cross-module agreement an enforced invariant rather than a convention, the same move as requiring the rotation rather than declaring it away. And the failure mode it produces is the right one: when #762 moves the cap, a stale rotation stops the scorer instead of quietly being reported beside freshly-computed aggregates. The candidate needs re-measuring at that point, which is exactly what the refusal says.

The declared cap is also echoed into the receipt's candidate_holdout block, so the number carries its own provenance rather than relying on the top-level label to vouch for it.

2 — the row scale, imported rather than restated

Right, and cheap to close: microcosm.calibrate already exports default_target_loss_scales, so _relative_errors now calls it instead of hand-rolling max(abs(target), 1.0). The quantity stays what the docstring says it is — signed and uncapped, feeding only the drift rows and win counters — but it can no longer drift from the aggregates printed beside it, and the docstring's "never carries a second copy" claim is now literally true rather than true-modulo-the-denominator. It vectorised naturally, which also removed the per-row Python loop over the registry.

Noted on math.isclose vs exact equality for worst_holdout_loss — agreed, and left as is.

Rebase onto #829

Six conflicts, one of which deserves flagging:

Verification (post-rebase)

battery/spec 404 passed; UK local + cross-grain + UC-capital lane 329 passed, 2 skipped; microcosm-data contract 228 passed; ci_test_groups.py --verify ok; ruff clean. GitHub reports the PR mergeable.

@juaristi22
juaristi22 merged commit 4c6cc58 into main Sep 1, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants