Plan: replace eFRS with Microplex UK record-count datasets

[MaxGhenis]

Microplex path to replace eFRS

This is the operational plan for replacing PolicyEngine's Enhanced FRS
(eFRS) with Microplex-produced UK datasets.

The plan follows the US eCPS replacement framing but uses UK-specific source
and target facts. Dataset names should be record-count tiers, not use-case
labels:

• mp-30k-uk: routine UK microsimulation tier, roughly current eFRS scale.
• mp-300k-uk: larger candidate tier for stronger regional/local support.
• mp-3m-uk: future parent universe for sparse/distilled UK tiers if source
  access and compute make it useful.

The long-run construction path should be hierarchical: build the largest
defensible UK candidate universe, then use L0 sparsity and post-fit pruning to
derive smaller tiers. The first mp-30k-uk candidate may be a directly sampled
or directly fused build, but that should be treated as an initial benchmark
candidate rather than the permanent construction contract.

Source architecture

The UK source analogs are:

US analogue	UK source	Role
CPS ASEC	FRS	core household, benefit-unit, income, benefits scaffold
PUF	SPI	tax-unit/top-income donor source
SCF	WAS	wealth, assets, debts, pensions, property
CEX	LCFS	consumption, expenditure, indirect-tax base
admin/local targets	HBAI, HMRC, DWP, ONS, local authority, constituency targets	calibration and validation

microplex-uk already has source adapters and benchmark scaffolding for FRS,
SPI, and WAS. The replacement work should push that spike into a production
pipeline.

Replacement gates

mp-30k-uk can replace eFRS as the routine UK dataset when all of these are
true:

1. Pinned baseline: every comparison records the eFRS H5 path, eFRS SHA256,
   policyengine-uk-data commit, policyengine-uk version, target source
   paths, and target source hashes.
2. Schema compatibility: passes an automated H5 contract check against the
   current policyengine-uk loader, including household, benefit-unit, person,
   and tax-unit IDs, joins, weights, periods, dtypes, and missing-value
   conventions.
3. Target loss: beats pinned eFRS on common supported UK target suites.
4. Protected families: no material regression versus eFRS on income tax,
   National Insurance, Universal Credit, child benefit, pension credit, housing
   support, disability benefits, household net income, wealth, rent, and
   council-tax-relevant variables.
5. Microsimulation benchmarks: passes a frozen UK reform suite covering income
   tax, National Insurance, Universal Credit, child benefit, pension credit,
   housing costs, disability benefits, and at least one Scotland/Wales/devolved
   variation.
6. Runtime: median runtime over the fixed UK benchmark suite is no more than
   1.25x eFRS for default release. A beta can be slower if explicitly labeled.
7. Artifacts: writes config, source versions, target hashes, score files, target
   deltas, record counts, nonzero weights, effective sample size, H5 size, and
   benchmark outputs.

Workstreams

1. Build first mp-30k-uk candidate

Deliverables:

• FRS scaffold.
• WAS wealth imputation through Microplex donor semantics.
• SPI tax/top-income donor integration.
• eFRS comparison on pinned PolicyEngine-UK targets.
• record-count, ESS, and H5 compatibility report.

Exit:

• dashboard shows mp-30k-uk versus pinned eFRS with target deltas.

2. Harden UK target registry

Deliverables:

• import UK calibration/validation targets into canonical Microplex target
  specs
• label target source, entity, scope, geography, period, and aggregation
• distinguish HBAI/disposable-income definitions from PolicyEngine model
  variables
• prevent source-scope confusion in tests

Exit:

• no release-suite target lacks source, scope, and entity provenance.

3. Add UK H5 compatibility checks

Deliverables:

• automated contract check for policyengine-uk H5 loading
• entity join and weight validation
• missing-value and dtype checks
• report usable by the dashboard

Exit:

• no UK candidate can be release-track without a passing compatibility report.

4. UK microsimulation benchmark suite

Deliverables:

• frozen manifest of UK reforms and periods
• eFRS baseline outputs and Microplex candidate outputs
• aggregate fiscal impacts, winners/losers, household net income, poverty/HBAI
  where applicable, and component deltas
• use PolicyEngine weighted series APIs; no manual weight handling

Exit:

• every mp-30k-uk candidate has a comparable report against eFRS.

5. Scale to mp-300k-uk

Deliverables:

• larger candidate universe using available UK sources
• L0 or post-fit pruning path back to mp-30k-uk
• local/geographic validation where available
• profiled build stages and disk guardrails

Exit:

• mp-300k-uk improves the frontier, or the dashboard clearly marks it as an
  experimental tier.

6. Publish and switch

Deliverables:

• policyengine-uk-data publication path for mp-30k-uk beta artifact
• policyengine-uk default-switch and rollback plan
• release notes explaining eFRS as incumbent comparator, not truth

Exit:

• PolicyEngine can load mp-30k-uk through the normal UK dataset interface and
  switch defaults only after gates pass.

[MaxGhenis]

Tracking issues filed:

• Build and score first mp-30k-uk eFRS replacement candidate #3 — Build and score first mp-30k-uk eFRS replacement candidate
• Add UK H5 compatibility checks for Microplex exports #4 — Add UK H5 compatibility checks for Microplex exports
• Build UK Microplex replacement dashboard gates #5 — Build UK Microplex replacement dashboard gates
• Harden UK target registry for Microplex calibration and validation #6 — Harden UK target registry for Microplex calibration and validation
• Add UK microsimulation benchmark suite for mp-30k-uk versus eFRS #7 — Add UK microsimulation benchmark suite for mp-30k-uk versus eFRS
• Scale UK Microplex to mp-300k-uk and derive smaller tiers #8 — Scale UK Microplex to mp-300k-uk and derive smaller tiers
• Publish and load Microplex mp-30k-uk beta artifact policyengine-uk-data#422 — Publish and load Microplex mp-30k-uk beta artifact
• Define PolicyEngine-UK switch and rollback path for mp-30k-uk policyengine-uk#1692 — Define PolicyEngine-UK switch and rollback path for mp-30k-uk