UK parity increment 4: port the weighted integrity gates (input-mass parity + QRF tail concentration) into the UK terminal battery (#578)

[juaristi22]

Where this sits

#578 ships a candidate only when it beats the incumbent on a frozen comparison register and the versioned invariant battery passes — the codified list of what the loss cannot see. The UK parity audit (#578 comment 5131554228, 2026-07-30) found the UK path carried essentially none of that battery and ranked an 8-item path to launchable parity.

Items 1–3 landed in #588 (merged 2026-08-01):

item	landed as
1. Freeze naming/lineage (frs, cps-transfer)	uk_runtime/release_identity.py
2. Standardize diagnostics (ESS, ratios, top-1%, per-geo pass rates)	uk_runtime/diagnostics.py (uk_weight_summary)
3. Consolidated terminal gates + degenerate sweep	uk_runtime/terminal_gates.py — six gates under one HMAC-attested batched report

This issue is item 4: port the weighted integrity gates. Items 5–8 have no tracking issue yet; item 6 is partly served by #349/#495 and item 7 by #144.

The UK now has the better container — versioned, signed, one batched fail-closed report — with fewer checks in it than the US. Increment 4 closes the two that were each purchased with a named incident.

What is missing, exactly

Verified on main @ 8ac0d7e: neither input_mass_parity nor any tail-concentration check appears anywhere under uk_runtime/, uk/, or the UK build tools. terminal_gates.py imports exactly three shared gates — export_surface_gate, weights_audit_gate, target_surface_gate — plus uk_release_input_coverage_gate.

Two things that look adjacent but are not substitutes:

• uk_release_input_coverage_gate is a presence check, not a level check. Its floor is minimum_nondefault_mass_share = 1e-06 (release_input_coverage_manifest.json, reviewed 2026-07-11). A column that ships one millionth of its reference mass passes. The Sparse-57k certified release zeroes untargeted input bases (IRA/HSA/SE-pension/childcare); CDCC and ALD reforms silently score ~$0 on bundle 4.18.8 #278 signature — a column that survives as a name while losing its base — is exactly what that floor lets through.
• uk_weight_ess_gate / uk_weight_ratio_gate bound the weight vector. Tail concentration is a property of |value| × weight per imputed column. A file with a perfectly healthy ESS can still ship a QRF column whose mass sits in a hundred records; the Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462 incident had no weight pathology at all.

Part A — input-mass parity

US implementation

piece	location
shared gate	packages/populace-build/src/populace/build/gates.py:1012 (input_mass_parity_gate)
totals helper	packages/populace-build/src/populace/build/us_runtime/input_mass.py:24 (us_input_mass_totals)
call site 1 — base frame vs certified reference release	tools/build_us_fiscal_refresh_release.py:4704 (_input_mass_reference_gate)
call site 2 — export support vs certified reference frame	tools/build_us_fiscal_refresh_release.py:4986 (_export_input_mass_gate)
thresholds	--input-mass-relative-tolerance 0.5, --input-mass-minimum-reference-total 1e9
escape hatch	--allow-input-mass-drift (records without failing; release builds leave it unset)
artifact	input_mass_parity.json, written before the batched raise

Semantics worth preserving verbatim: a zero candidate total fails at any tolerance ("a zeroed input base is the same failure as an absent one"); candidate-only columns are reported and never fail; near-zero reference columns are skipped because relative drift on them is meaningless; unused reviewed exclusions are reported so the register cannot rot.

The reference choice is load-bearing and was itself adjudicated. #327: comparing a calibrated export against the raw base flagged correct, target-aligned upward alignment of under-reported PUF income as failure, so the live default became a certified reference release while the #278 signature (candidate == 0, or candidate ≪ reference) still fails.

The UK port

us_input_mass_totals is already country-agnostic — it reads only frame.schema (person_id_column, group_entities, id_column, membership_column). Promote it to a shared populace.build.input_mass (or add a thin UK wrapper); do not fork it.

The column surface exists: PolicyEngineUKCoverageEngine.variables() (uk_runtime/release_input_coverage.py:494) returns the effective loader-override surface — 866 engine-known persisted variables per efrs_parity_reference.json.

Open design question — which reference. Three candidates, none free:

1. enhanced_frs_2023_24.h5 (sha 584ae33d…, revision 655dd07e…) — the incumbent the coverage contract already measures against, and the true analog of the US eCPS frozen reference. It is UKDS-licensed and never ships, so the reference must be frozen: extend tools/build_uk_efrs_parity_reference.py to emit weighted per-column totals alongside the nonzero shares it already computes. Frozen also gives the US doctrine property — a candidate cannot move the bar by choosing its reference.
2. The certified compact populace_uk_2023.h5 (sha f17306cc…) — already byte-verified (verify_certified_uk_candidate, tools/build_uk_national_dataset.py:159) and already loaded in-process at national_build.py:415, so no new artifact is needed. But the HMRC/SPI stage intentionally moves mass on the restored surface: replace_zero_weight_spi_support allocates 50% of national household prior mass to the rebuilt SPI channel. A naive staging-vs-compact comparison reproduces the Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 failure mode on the restoration surface.
3. The staging H5 — the right reference for everything downstream of the seam (the UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495 rowwise arm, and the item-8 exact-k selections, where the Sparse-57k certified release zeroes untargeted input bases (IRA/HSA/SE-pension/childcare); CDCC and ALD reforms silently score ~$0 on bundle 4.18.8 #278 signature actually lives).

Recommendation: (1) for the seam gate and (3) for the downstream gate, mirroring the US pair. If (2) is chosen instead, the restoration surface needs either loss-only semantics or a register derived from the manifest's restoration_evidence / replay declarations rather than hand-listed.

Two things to settle before writing code:

• Licensing. efrs_parity_reference.json currently commits unweighted nonzero shares. Weighted totals are a different disclosure class. Confirm the UKDS EUL permits committing aggregate weighted totals derived from the pinned artifact before that file is written.
• The floor. 1e9 is a US-dollar figure against a 337k-record pool. UK totals are GBP against 28.8M household mass; the floor must be measured, not inherited.

Part B — QRF tail concentration

US implementation

piece	location
shared gate	gates.py:2365 (tail_concentration_gate)
UK-relevant surface derivation	tools/build_us_fiscal_refresh_release.py:5041 (_qrf_imputed_source_outputs) — derived from us/source_stages.json where operation.kind == "fit_weighted_qrf", so a new QRF stage is covered the day the manifest declares it
wrapper	tools/build_us_fiscal_refresh_release.py:5056 (_qrf_tail_concentration_gate)
thresholds	top_k=100, max_top_share=0.75, min_nonzero_records=500, sparse cutoff nonzero_share <= 0.05
register	--qrf-tail-concentration-exclusions (JSON, path + sha recorded); escape hatch --allow-qrf-tail-concentration
artifact	qrf_tail_concentration.json, written before the batched raise

The incident (#462): the Build M puf_tax_detail weighted QRF broadcast a donor-tail point mass (a repeated $594,484 ceiling value) onto non_sch_d_capital_gains, so 100 of 2,295 carriers held 89% of the shipped $74.6B — 7.3× the SOI dollar target — while the paired returns-count target hit exactly. Support clipping cannot see it (every draw is inside the donor's realized range) and count targets cannot either. Weighted-mass concentration is the only tell.

The UK is exposed to the same mechanism today

The UK national build runs two weighted QRFs off a private licensed donor:

• populace.fit.QRF via uk_runtime/spi_income.py:999, donor = SPI PUT 2022-23 (put2223uk.tab, UKDS SN 9422).
• Declared in uk/hmrc_income_source_stages.json as fit_weighted_qrf_stage1 and fit_weighted_qrf_stage2, both carrying an explicit outputs list and validated by uk_runtime/hmrc_source_contract.py:475,614.
• Stage-1 surface: 18 columns (SPI_INCOME_QRF_OUTPUT_COLUMNS, uk_runtime/spi_support.py:102) — eight income/relief components plus ten HMRC SPI leaves. Stage-2 surface: FRS_ONLY_SPI_FILL_PERSON_COLUMNS (spi_support.py:133) — pension and reported-benefit behaviour replaced on SPI support rows.

So the manifest-driven derivation the US uses is available today, without waiting on the full uk/source_stages.json that #156 would add. Build the surface from the fit_weighted_qrf_stage* operations' declared outputs, not a hand list.

The sparsity filter does not transfer. US_QRF_SPARSE_NONZERO_SHARE_MAX = 0.05 encodes a specific premise: a sparse imputed column sitting on a mostly-observed CPS spine. On the UK staging pool the SPI channel is 432,779 of 1,157,739 persons (#495's committed staging record), and the QRF outputs are donor-drawn for that entire channel — several will sit far above a 5% nonzero share and be silently skipped, making the gate vacuous exactly where the risk is. Proposal: check every declared QRF output regardless of density, and let min_nonzero_records remain the sole thinness guard (it already ensures top_k is a strict subset of carriers). Any narrowing of that surface must be justified in the manifest, not in a constant.

Thresholds must be measured, not inherited. The US numbers are calibrated to the US incident. The UK has no incident yet, so increment 4 owes a measurement pass — the same discipline that set UK_MAX_TO_MEDIAN_WEIGHT_RATIO to the incumbent's exact 1151.2542195939373 with no discretionary headroom.

Wiring

One place, following the increment-1 pattern:

• Add both gates to the evaluators list in uk_terminal_gate_report (uk_runtime/terminal_gates.py:1212), using the optional-evidence pattern at :1262 so a path with no reference evidence omits the gate rather than inventing a pass — the same choice tools/build_uk_national_dataset.py:173 already makes for the parity trio.
• Thread every new threshold and register through _terminal_gate_policy_payload (:336) so policy_sha256 covers them. A threshold outside the policy hash is not attested.
• Add an evidence_sha256 family per new evidence object (:1299), alongside release_dataset / hmrc_spi_income / release_parity.
• Bump UK_TERMINAL_GATE_SCHEMA_VERSION (and the attestation version if the payload shape moves), since consumers key on gate names.
• Batched evaluation is already correct: _evaluate_gate (:1137) fails closed on a raising evaluator and every gate still runs, the report is written at national_build.py:442, and the raise follows at :443. Do not add a second raise.
• Second call site for the downstream arms (UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495 rowwise, item 8 exact-k): calibrated candidate vs staging, mirroring _export_input_mass_gate.

Deliverables

0. Measurement pass first. Weighted per-column totals for the certified compact, the pinned eFRS incumbent, and the current staging candidate; top-100 weighted mass share and carrier count for every declared QRF output on both. Post the numbers on US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578 before any threshold is written.
1. Shared input_mass_totals (promoted, not forked) + a uk_input_mass_parity_gate wrapper returning GateResult(name="input_mass_parity").
2. uk_qrf_tail_concentration_gate with its surface derived from the source manifest's fit_weighted_qrf_stage* outputs.
3. Both wired into the attested batched report per above, with thresholds set from step 0.
4. Committed reviewed-exclusion registers under the universal discipline: mandatory non-empty reason, dormant entries reported, stale entries fail (a column now inside the threshold is a register that rotted).
5. Mutation tests to the increment-1 standard: removing either gate must fail a committed test; a zeroed column, a 99.9%-loss column, and a concentrated column must each fail by name; a crash inside either evaluator must append a named failure line rather than mask the pending ones (SSI delivery gate raises before diagnostics write and outside the #437 batch — a failed run destroys its target-surface evidence #547).

Acceptance criteria

• Measurement numbers posted on US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578; thresholds set at the measured boundary with no discretionary headroom, and the derivation recorded next to the constants.
• Both gates appear in the UK terminal gate report by name and are bound into policy_sha256.
• Reference identity (filename, revision, sha256, vintage) recorded in the report for the parity gate.
• A candidate that zeroes any material persisted input column fails; a candidate that keeps every column within tolerance passes.
• A candidate whose QRF output concentrates beyond the threshold fails; a stale exclusion fails.
• Evidence artifacts written before the raise; a red run leaves the diagnosis on disk.
• Legacy behaviour byte-identical on paths that supply no new evidence.

Non-goals

Item 5 (delivered take-up gates on UK administrative anchors), item 6 (OA ladder wiring — #349/#495), item 7 (raw-only rebuild — #144/#156), item 8 (UK exact-k suite — #355). No threshold loosening, no per-target knobs, no new escape hatch beyond a diagnostic recorder that release builds must leave unset.

Refs

#578 (epic and acceptance rule), #578 comment 5131554228 (the audit and the 8-item backlog), #588 (increment 1), #278 (mass-parity incident), #327 (reference-choice adjudication), #462 (QRF tail incident), #481 (upstream QRF donor-weight defect), #396 (effective-mass coverage semantics), #547 (degraded-mode batched failure), #144, #156, #349, #355, #495.