UK local solve through the public calibrate front door, increment 2 of #612 - #634
Open
juaristi22 wants to merge 5 commits into
Open
UK local solve through the public calibrate front door, increment 2 of #612#634juaristi22 wants to merge 5 commits into
juaristi22 wants to merge 5 commits into
Conversation
The pre-ladder stacked areas x households cross-product solve had zero production callers (tests only), was superseded by the rowwise ladder path, and structurally can never route through the public calibrate() front door this increment migrates onto — its weight vector is longer than any entity table. Removed: solve_stacked_local_weights, the stacked doctrine route, local_runner's candidate build path, and the stacked matrix/long-weights helpers in local_geography. Everything deleted was UK-specific uk_runtime code; country-agnostic populace-calibrate is untouched. Kept: solve_prepared_local_weights and StackedLocalSolveResult (the rowwise path consumes them until the calibrate() migration lands), the reviewed doctrine constants, past_cap_census, and align_area_targets. The uniform-target-surface check moved beside its only remaining consumer in local_rowwise. sort_households_by_id retires with its only consumer: positional-assignment safety on the surviving paths rides on the Frame kernel's sorted-group-id invariants and validate_uk_national_frame. The SPI lineage test now asserts the anti-doubling property directly on source_household_key instead of through the retired long-weights representation. Refs #612, #578. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rowwise release path's clone result was the last UK shadow carrier on
this arm: three raw DataFrames plus a weight-kind string and a mass log,
kept off Frame only by the artifact's shared clone_index column name on
all three entity tables (Frame's flattening rule refuses it). Clone
indices now land on per-entity {entity}_clone_index columns in memory, so
the result carries a validated UK national frame — typed household
weights under the #501 kind-fence chain, the mass log, and time_period as
frame metadata — and the cross-clone link-alignment fence reads the
per-entity names.
The export boundary is unchanged: the writer materializes engine tables
from the frame (typed weights authoritative) and renames the per-entity
columns back to the artifact's single clone_index name per table, in
place, preserving column order; the ladder-clone test pins the written
schema (column lists, clone_index position, attrs) so a carrier change
can never leak into the H5. The candidate tool asserts the kernel kind
transition explicitly and hard-constructs the calibrated frame with the
refreshed weight column and appended calibration record — the #618 stage
idiom — until the calibrate() front-door migration owns both. The
crosswalk route keeps its table-carrying result; its carrier is a
recorded follow-on.
The pool-lineage fixture now ascends its household ids: every production
producer is a Frame load, which guarantees sorted group ids, and the
carrier lets Frame raise on unsorted input rather than sorting rows
itself (the FRS-2024-25 bug class).
Refs #612, #495.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt (#612 inc 2) clone_uk_dataset_with_ladder_geography refuses the duck-typed in-memory carrier: an in-memory input must be a populace Frame (weight kind and mass log from the typed weights and frame, nothing defaultable), and only an H5 path keeps the documented attribute-less DESIGN semantics. The crosswalk route keeps its duck input — its carrier swap is the recorded follow-on. Test ducks became real frame constructions; the negative-weight guard is exercised at the raw-table entry, where a kernel-refused input cannot mask it. UK_LOCAL_MAX_WEIGHT_RATIO carries the populace#493 adjudication record: the bound stays 100.0 (vs the US design bound 5.0, realized 4.994, and the exact-k ladder's 20.0) because tightening without measured fit would be an arbitrary cutoff; the revision path remains the #495 increment-6 calibrated-candidate review through this constant and its pinned test. national_frame's docstring records the household_weight column-drop assessment: the column is a materialized export contract regenerated by engine_tables, the carrier's own validation makes it load-bearing, and the drop is sequenced behind the #611 consumer half and the reader moves. Refs #612, #493, #495. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inc 2) The UK local solve was the last consumer of populace.calibrate's private internals (_optimize, _search_l0_lambda_for_budget, _torch_constraint_matrix) — it hand-assembled a COO/CSR system because it had no Frame at the solve site. With the ladder clone Frame-native, the doctrine solve now expresses the area x metric surface as a declarative TargetSet (the metric vector as the measure, area membership as the filter, both closing over the problem's own arrays) and calls the same public calibrate() entry the US production paths use. The kernel enforces the CALIBRATED kind transition and mints the calibration mass record; calibrate() gains an additive mass_reason parameter so a free-mass record can carry domain context (the bound target families) instead of the generic reason, and the record now declares the realized factor. The hand-assembled UKRowwiseLocalMatrix stays as the fail-closed surface definition and evidence substrate: builder refusals (uncovered areas, unreachable nonzero targets), the per-area support summary, the labelled diagnostics frame, and the populace#492 past-cap census all keep their pre-migration vocabulary — and a differential test pins the declarative compile to the COO assembly's matvec row for row, so the two representations provably derive from the same numbers. The doctrine's constants ride into the front door explicitly and are pinned by a forwarding test (calibrate's own defaults differ, so default-drift would change behaviour). solve_prepared_local_weights, StackedLocalSolveResult, the optimizer floor (zero base weights are refused; tiny positives are legal seeds), and the hand-minted rowwise_calibration_mass_record retire; local_solver.py dissolves into local_rowwise.py. Refs #612, #493, #495. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… precision (#612 inc 2) The review's blocker: the production staging input's household table carries a candidate-tier clone_index (SPI/pool lineage) that the synthetic fixtures never had. The pre-Frame clone silently overwrote it; the Frame carrier left it beside the new per-entity name, and the writer's rename then produced duplicate column labels — a raw pandas crash after the full solve. The clone now replaces the inherited column explicitly at entry (same artifact semantics as the pre-Frame writer, now documented), and the reserved {entity}_clone_index names fail closed instead of being silently overwritten. Regression tests cover both, on the production input shape. Also from the review: the candidate tool refuses _CONSERVE_MASS upfront (the manifest's calibration_mass_change block reads the kernel's free-mass record; flipping the constant used to brick the tool after the solve); calibrate() records mass_reason in CalibrationResult.options so manifest consumers see the declared reason, not only the mass log; and the doctrine solve reports initial_loss in float64 from the compiled estimates again (the trajectory head is a float32 optimizer value — the reported figure had drifted at the eighth digit). Refs #612. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Increment 2 of #612, on the #618 Frame carrier now on main: the UK local/rowwise solve moves onto the public
populace.calibratefront door — the same entry point the US production paths use (build_us_acs_local_release.py,build_us_fiscal_refresh_release.py, the exact-k ladder) — so UK weights take the kernel-enforcedWeightKind.CALIBRATEDtransition and the kernel mints the mass record. All six roadmap commits have landed; draft pending the final CI pass and the compile-cost probe note.Commit roadmap
areas x householdscross-product solve had zero production callers (tests only), was superseded by the rowwise ladder path, and structurally can never route throughcalibrate()— its weight vector is longer than any entity table. Removed:solve_stacked_local_weights, the stacked doctrine route,local_runner's candidate build path, and the stacked matrix/long-weights helpers inlocal_geography. Kept:solve_prepared_local_weights+StackedLocalSolveResult(the rowwise path consumes them until commit 3), the reviewed doctrine constants,past_cap_census, andalign_area_targets. Everything deleted was UK-specificuk_runtime/code — nothing in country-agnosticpopulace-calibrateis touched. The uniform-target-surface check moved beside its only remaining consumer inlocal_rowwise.sort_households_by_idretires with its only consumer; positional-assignment safety on the surviving paths rides on the Frame kernel's sorted-group-id invariants andvalidate_uk_national_frame.{entity}_clone_index) so Frame's column-uniqueness holds; the written H5 keeps the legacy schema (singleclone_indexper table — the writer renames at the export boundary), asserted by an old-vs-new payload test on synthetic tables.calibrate()(landed). Area×metric targets as a declarativeTargetSet(area membership as filters,Target.metadatacarrying area/metric labels), today's solver constants passed explicitly and pinned (epochs 512, lr 0.15, ratio 100.0, cap 10.0), kernel-minted mass record reconciled with the writer's chain-currency fence, past-cap census keys unchanged. The three privatepopulace.calibrate.solveimports die here. A differential test pins the declarative compile to the COO assembly's matvec row for row; the kernel record now declares the realized mass factor (the hand-minted predecessor left it None). Compile-cost probe at real scale: recorded when the next credentialed candidate build runs.dataset: Anysurface (landed) (rowwise_datasetnarrowing toFrame | str | Path, the candidate tool).household_weight-drop assessment (blocked by UK carrier swap, increment 1 of #612: carry the national build on populace Frames (stacked on #610 + #617) #618's own carrier contract and One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578) #611-fenced readers; documented, deferred).Fences
terminal_gates.py,weighted_integrity.py,release_input_coverage.py,national_build.py).load_uk_national_frame's signature stable for the candidate tool here.firm_generation.py's two private imports are out of scope (follow-up issue; its per-target loss weights and external torch matrix need their own public-seam design).Refs #612, #578 (item 2.5 ladder), #493, #495.
🤖 Generated with Claude Code