UK dataset build: full migration from policyengine-uk-data (master epic)

[juaristi22]

Goal

Migrate the UK dataset build fully from policyengine-uk-data into microcosm, at parity, then archive policyengine-uk-data (the policyengine-us-data precedent). End state: policyengine-uk consumes only microcosm-built datasets — national default, sparse default, and the local variant — calibrated exclusively from chronicle facts.

This epic supersedes #144, which tracked the same goal as a flat backlog.

Architecture invariants

• Frame is the carrier through every stage; calibration is rowwise via the microcosm.calibrate front door.
• Country package = pure JSON specs (build/uk/), executed by shared runtimes; Python in uk_runtime/ migrates toward specs (Move country build choices into declarative specs #159).
• Target values come only from pinned chronicle consumer artifacts; computed targets (uprating, rate×count, splits) are explicit microcosm-side declarations.
• Scope ruling carried from Migrate remaining policyengine-uk-data backlog before archive #144 (2026-07-24): the build is based solely on raw microdata, ledger facts, and the rules engine (plus inflation/aging updates) — no inheritance of processed artifacts whose generating code stays behind in uk-data.
• No stage ships degraded: fail-closed plan, closed-world gate entries, evidence_absent over invented passes.
• Every build attempt — including failed rungs and discarded candidates — appends a hash-chained Logbook row (Chronicle: one append-only trace for every build attempt at every rung #628), so the migration's intermediate-build history stays reconstructible.
• Never import policyengine_uk_data — port definitions, don't wrap them.

Phases

Phase	Contents	Milestone
P0	Unblockers (A)	Fresh national rebuild passes armed gates; schema-4 verification lands
P1	Facts wave 2 (B) + target references (C) + calibration seam (D)	First microcosm-calibrated national release, on the certified spine, from chronicle facts only
P2	Raw ingest + imputations (E) + uprating (F)	Spine built in microcosm from raw sources; FRS 2024-25 refresh
P3	Facts wave 3 (B) + local areas (G)	UK local variant published (constituency + LA weights)
P4	Sparse default (H) + downstream integration (I)	~50k L0 national default; policyengine-uk consumes microcosm datasets
P5	Decommission (J)	policyengine-uk-data archived

Workstreams

• A — Unblockers: Schema-4 gate-battery verification in the release contract (#611 A3) #663, National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630, UK parity increment 4: port the weighted integrity gates (input-mass parity + QRF tail concentration) into the UK terminal battery (#578) #609, Finish the #612 tails: drop the carrier household_weight column; narrow the crosswalk clone entry #654, Move firm_generation off populace.calibrate's private optimizer internals #636, Record every UK build attempt in Logbook #666 (UK Logbook adoption), UK CI lane (to be filed)
• B — Facts to chronicle: UK source packages, wave 2: national fiscal/benefit families + uk_national target profile chronicle#133, UK source packages, wave 3: back the uk_local_geography profile (constituency + local authority) chronicle#134, UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup chronicle#135
• C — National targets from facts: Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate #622
• D — National calibration seam: Calibrate the UK national build from Ledger-backed targets #623
• E — Raw-source ingest + imputations: Port UK compact source stages and 2024-25 refresh from uk-data #145 (absorbs Migrate remaining policyengine-uk-data backlog before archive #144 remnants; in-flight children: Impute capital gains amounts from the published joint distribution #560, UK: household-wealth imputation stage with cash / S&S ISA split #181, Add UK bus-spending imputation plan and DfT calibration targets #153)
• F — Uprating / multi-year: Decide UK panel, public transfer, and other product-scope leftovers from uk-data #148
• G — Local areas (rowwise): UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495 (children: Publish a Populace UK local variant with rowwise H5 and long weights #146, Port UK local target surface and credibility gates from uk-data #147; doctrine: Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic #492/max_weight_ratio anchors differently per arm: 5x vs design (dense) but ~25x effective (sparse refit re-anchors) — declare one stretch contract #493)
• H — Sparse national default: UK small national default: L0-selected ~50k-household file from the 535k pool #355
• I — Release + downstream integration: tracking issue to be filed when the workstream spins up (includes a policyengine-uk child issue for loader consumption)
• J — Decommission: tracking issue to be filed when the workstream spins up

Related, non-blocking: #365, #31, #552/#364/#467, #180, #325, Logbook epics #637/#641 (workstream H publication surface), #661 (diagnostics carry chronicle package IDs — workstream D), naming cleanups #639/#640.

Carried-over uk-data backlog

policyengine-uk-data stayed active after #144's June bookkeeping and accumulates new issues — 5 open as of 2026-08-12: PolicyEngine/policyengine-uk-data#438 (stale public-transfer feed, state pension ≡ £0 → workstream F/#148), PolicyEngine/policyengine-uk-data#448 (Scottish/Welsh council tax +20–30%, WAS owned_land instability → E and C/D), PolicyEngine/policyengine-uk-data#450 (UC deduction attributes imputation → E, new stage), PolicyEngine/policyengine-uk-data#452 (UC caseload ~40% below admin counts → D/E), PolicyEngine/policyengine-uk-data#453 (unsourced mortgage target → B, already a recorded chronicle blocked source). Dispositions live in the roadmap doc's Appendix B; workstream J re-sweeps the live list at decommission time.

Sequencing notes

• C/D deliberately run on the existing certified spine first: prove facts→targets→calibration→gates at parity before any ingest work, then E swaps the spine underneath a fixed harness.
• E (microcosm-side) and wave 3 (chronicle-side) parallelize; G needs both.
• H needs only D + schema-4 verification — the exact-k machinery is country-agnostic.
• The incumbent keeps releasing until I completes; every incumbent refresh widens the parity bar.

[juaristi22]

The pipeline, and where each piece stands

Status diagram for this migration, in the style of microcosm-dynamics#392. One node per pipeline piece, colored by state; updated as work lands (statuses as of 2026-08-12).

flowchart LR
  subgraph CHRON["1 · Facts — chronicle"]
    W1["<b>Wave 1</b><br/>149-target national surface"]
    W2["<b>Wave 2</b><br/>national families<br/>housing PR + uk_national profile"]
    W3["<b>Wave 3</b><br/>local geography packages<br/>constituency + LA"]
    SUITE["<b>UK consumer artifact</b><br/>suite wiring"]
  end

  subgraph TGT["2 · Targets — microcosm"]
    REFS["<b>uk_target_references</b><br/>+ mapping + compile-parity<br/>#622"]
  end

  subgraph SPN["3 · Spine + imputations"]
    SPINE["<b>National spine</b><br/>today: certified eFRS H5<br/>from uk-data — to be replaced"]
    FRS["<b>Full raw FRS ingest</b><br/>~14 tables; 2 read today<br/>#145"]
    SPI["<b>SPI donor + HMRC ODS</b><br/>income stages"]
    IMPA["<b>CGT · wealth/ISA</b><br/>PRs #560 · #181"]
    IMPB["<b>8 more imputation stages</b><br/>consumption · VAT · services/NHS ·<br/>salary sacrifice · student loans ·<br/>childcare · disability · UC deductions"]
    UPR["<b>Uprating / multi-year</b><br/>#148"]
  end

  subgraph GEO["4 · Geography"]
    OA["<b>OA ladder</b><br/>full UK"]
    CLONE["<b>Rowwise clone</b><br/>+ area assignment"]
  end

  subgraph CAL["5 · Calibration"]
    NATCAL["<b>National calibration seam</b><br/>#623"]
    LOCCAL["<b>Rowwise local calibration</b><br/>1 of 7 families bound;<br/>LA targets unbound"]
  end

  subgraph GATE["6 · Gates + record"]
    BATTERY["<b>Gate battery</b><br/>13 gates, signed schema-4"]
    SCHEMA4["<b>Schema-4 verification</b><br/>PR #663"]
    REBUILD["<b>Fresh rebuild passes gates</b><br/>#630: weight_ratio<br/>1590 vs 1151"]
    ARM["<b>Weighted-integrity<br/>gates armed</b><br/>#609"]
    LOGBOOK["<b>Logbook: every attempt<br/>recorded</b><br/>#666"]
  end

  subgraph REL["7 · Products"]
    NAT["<b>National release</b><br/>facts-calibrated"]
    SPARSE["<b>Sparse exact-k default</b><br/>~50k of 535k · #355"]
    LOCVAR["<b>Local variant published</b><br/>#146"]
  end

  subgraph DOWN["8 · Downstream"]
    PEUK["<b>policyengine-uk consumes</b><br/>microcosm datasets"]
    VALID["<b>Reform-validation suite</b><br/>#365"]
    ARCHIVE["<b>Decommission</b><br/>uk-data archived"]
  end

  W1 --> REFS
  W2 --> REFS
  SUITE --> REFS
  W3 --> LOCCAL
  REFS --> NATCAL
  SPINE --> NATCAL
  UPR --> NATCAL
  FRS --> IMPA
  FRS --> IMPB
  SPI --> SPINE
  IMPA --> SPINE
  IMPB --> SPINE
  OA --> CLONE
  SPINE --> CLONE
  CLONE --> LOCCAL
  LOCCAL --> LOCVAR
  NATCAL --> REBUILD
  BATTERY --> REBUILD
  ARM --> REBUILD
  REBUILD -.-> LOGBOOK
  REBUILD --> NAT
  SCHEMA4 --> NAT
  NAT --> SPARSE
  NAT --> VALID
  NAT --> PEUK
  SPARSE --> PEUK
  LOCVAR --> PEUK
  PEUK --> ARCHIVE
  VALID --> ARCHIVE

  classDef done fill:#2f6f52,color:#fff
  classDef flight fill:#b8721a,color:#fff
  classDef blocked fill:#8b3a3a,color:#fff
  classDef transitional fill:#4a5d78,color:#fff
  classDef todo stroke-dasharray:4

  class W1,SPI,OA,CLONE,BATTERY done
  class W2,IMPA,LOCCAL,SCHEMA4 flight
  class REBUILD blocked
  class SPINE transitional
  class W3,SUITE,REFS,FRS,IMPB,UPR,NATCAL,ARM,LOGBOOK,NAT,SPARSE,LOCVAR,PEUK,VALID,ARCHIVE todo

Loading

🟩	Done — merged and exercised
🟧	In flight — open PR, or partially landed
🟥	Blocked — fails today; needs adjudication or repair
🟦	Transitional — works today but exists to be replaced (the incumbent-built spine)
⬜	Not started

The critical path runs through boxes 2→5: uk_target_references (#622) → the national calibration seam (#623). Every product node waits on that chain, and it deliberately lands on the transitional spine first — proving facts→calibration→gates at parity before the raw-ingest work (box 3) swaps the spine underneath. The red box (#630) blocks release assembly independently of everything else.

Phase milestones (from the epic body): P1 = box 5's national seam calibrating from facts on the certified spine · P2 = box 3 replaces the spine · P3 = boxes 1(W3)+5 light up the local variant · P4 = boxes 7–8 · P5 = archive.

Maintained by @juaristi22 — statuses re-checked against the linked issues on each update.