Postmortem: how Universal Credit ran 40% under administrative counts for six months

[MaxGhenis]

Universal Credit aggregates in shipped UK datasets ran 35–55% below administrative counts from late February to mid-August 2026, across three artifact lines, without any alarm firing. This is the full trace, with measurements and links. It is a blameless postmortem: every contributing change was reasonable on its own terms, and the failure was systemic — nothing watched the output.

Impact

• The February–April enhanced_frs_2023_24.h5 artifacts carried 4.1–4.2m UC benefit units and ~£51bn UC spend against ~6.7m households (DWP, 2025 average) and £75.8bn (OBR).
• The policyengine.py certified UK default (populace_uk_2023, Microcosm) carried 2.17m (72aeefc, June) then 3.26m (dd68c73, the current certification candidate in policyengine.py#491) — its calibration surface has no UC targets at all (Certified UK artifact carries less than half of admin UC caseload (3.26m vs ~6.7m; £36.9bn vs £75.8bn) microcosm#701).
• The UC deductions module (policyengine-uk#1815, built for WPI Economics and JRF-style floor analysis) validated per household while every count and cost inherited the caseload: 1.9m deducting households / £1.2bn measured against DWP's 3.3m / £2.0bn (UC caseload runs ~40% below administrative counts, capping aggregate reform estimates #452).
• The working paper documenting that module shipped its abstract with a "~40% short" caveat that was, in fact, a property of a stale artifact pin.

Measured timeline

All measurements: weighted benefit units with universal_credit > 0 at 2025, annual basis, policyengine-uk 2.90, HF revisions of policyengine/policyengine-uk-data-private. (Aggregates only.)

date	event	UC benefit units	UC spend
Nov 2025	a local calibration log shows OBR UC count targets (1.6m + 4.7m) binding	6.32m (log)	£74.4bn (log)
2026-02-18	#269 merges: the targets registry rework	—	—
2026-02-24	v1.40.3 upload e00e9c87	4.19m	£51.3bn
2026-03-31	v1.45.8 upload 9a4709db	4.08m	£51.0bn
2026-04-13–16	#333 / #344 land: CI heartbeat + smaller builds so publication completes on hosted runners	—	—
2026-04-17	v1.50.6 upload 625c8e2d	6.42m	£77.6bn
2026-04-18	v1.52.2 upload 004f49df	6.34m	£76.9bn
2026-05-20	v1.55.10: last content change of enhanced_frs_2023_24.h5 (sha 584ae33d…)	6.40m	£77.6bn
2026-06-13/19	policyengine.py certified default flips to Microcosm populace_uk_2023 (72aeefc, then dd68c73)	2.17m / 3.26m	£24.0bn / £36.9bn
2026-07-26 → 08-18	v1.56.14/.16 releases re-upload the frozen May bytes; the live build line is now enhanced_frs_2024_25.h5	6.40m (frozen)	£77.6bn (frozen)
2026-08-17	#458 merges (this issue's target fixes)	—	—
2026-08-18	v1.56.16 enhanced_frs_2024_25.h5: post-fix live line	6.35m (target 6.7m)	£75.3bn (target £75.8bn)

What happened, in three layers

Layer 1 — the registry rework dropped one target family and mis-set another (#269, February)

#269 replaced ~700 lines of ad-hoc targets with the structured registry — a clear improvement, and the foundation everything since builds on. Two UC-specific regressions rode along, both verified by code archaeology:

1. The OBR UC caseload targets stopped being emitted. The dispatch branch for obr/universal_credit_jobseekers_count / non_jobseekers_count was carried into build_loss_matrix.py, but the new sources/obr.py never produced those targets (grep of the Replace ad-hoc targets with structured registry and source modules #269 tree: zero emission sites). The dispatch was dead code from day one — which is exactly why the drop was invisible. It stayed dead until Fix Universal Credit caseload targets #458 deleted it.
2. The family-type targets arrived with an inert undercount adjustment. undercount_relative = 1.27921 / sum(_UC_BY_FAMILY_TYPE.values()) divides what looks like millions by a sum in thousands: ×1.0002 instead of the intended-looking ×1.21. The four targets therefore encoded 6.096m rather than ~7.37m, and even a perfect calibration to them would have landed ~13% under administrative counts. This survived until Fix Universal Credit caseload targets #458.

Layer 2 — builds stopped completing, and releases shipped anyway (February–April)

The February and March artifacts measure ~4.1m — far below even the 6.096m the mis-set targets encoded, which means the targets were not binding at all in those builds. The best-supported mechanism is in the repo's own changelog: 1.50.3 "reduce OA cloning in GitHub Actions dataset builds so publication completes on hosted runners" and 1.50.4 "CI heartbeats … so dataset release builds do not get canceled while constituency target matrices are being prepared" (also #333). Dataset builds were being canceled mid-calibration on hosted runners, and the release step shipped what existed. The artifact healed to 6.42m at v1.50.6 (17 April) — immediately after #344 made builds fit the runners — with no UC-related target change in that window. (February CI logs have aged out, so this layer is labelled a strongly-evidenced hypothesis rather than a proven mechanism; #456's silent year-resolution drops are a secondary candidate.)

Layer 3 — consumers pinned the broken vintage, and the fixed file froze (April–August)

• enhanced_frs_2023_24.h5 last changed content on 20 May (v1.55.10). Every release since re-uploads the same bytes; the live build line moved to enhanced_frs_2024_25.h5. A version number advancing while the artifact stays frozen looks exactly like a fix shipping when it isn't.
• The UC deductions validation (policyengine-uk#1815) and the policyengine.py bundle pinned the February artifact (e00e9c87) into August — four months after the April repair existed.
• Meanwhile the certified policyengine.py default moved to Microcosm's populace_uk_2023 (June), which won its certification rotation on a 148-target surface containing no benefit targets — the winner was selected on a surface blind to the thing that was broken (Certified UK artifact carries less than half of admin UC caseload (3.26m vs ~6.7m; £36.9bn vs £75.8bn) microcosm#701).

Why nothing fired

• Per-household statistics divide the caseload out. The deductions module validated beautifully — incidence 47%, mean deduction £66/£50, FRR gain £421 vs the government's £420 — on a base 40% low. Ratios hid levels.
• An absent target is invisible to target diagnostics. Dashboards show fit for targets in the surface; neither the dropped OBR family (uk-data) nor the never-added UC family (Microcosm) could show as red.
• Failure modes were silent by construction: dead dispatch code masking a dropped emission; a units bug inside an adjustment factor; canceled builds still publishing; frozen files re-uploaded under new version numbers; targets dropped without error when a year fails to resolve (Calibration loss under-penalises overshoots and silently drops targets #456).

What is fixed, in flight, and remaining

Fixed: #458 (inert undercount removed; explicit dwp/uc/households totals 6.7m/7.2m derived from the deductions statistics Table 1; dead OBR dispatch deleted) — the post-fix live line measures 6.35m / £75.3bn, deducting households 3.02m / £1.86bn / 47.6% incidence.

In flight: the Microcosm chain — chronicle#184 (DWP UC facts, entity-explicit as benefit units), microcosm#727 (contract refresh adding dwp.uc.households), microcosm#729 (the national calibrate step: an activated reference that cannot resolve aborts the build). Then a populace_uk rebuild + recertification replaces the 3.26m certified default.

Remaining: refresh _UC_BY_FAMILY_TYPE to a current Stat-Xplore month (the stale 6.096m sum still pulls against the 6.7m total — the last ~5% on the live line); #456 (loss shape + silent drops) and #457 (dwellings-vs-households, ~+2.8m total households) as already scoped; and re-validating the deductions documentation on the 2024-25 line.

Lessons

1. Watch outputs, not just targets. A standing admin-truth benchmark (the Scorecard direction: OBR/DWP counterparts) would have turned every layer of this into a red number within days. Target diagnostics cannot catch an absent target.
2. Silent failure is the enemy. Emissions with dead dispatch, canceled builds that publish, targets dropped on unresolvable years — each should be a loud error. #729 makes unresolvable activated references abort; the same standard belongs everywhere in the build.
3. Version numbers must not outrun artifacts. If a release re-uploads frozen bytes, the manifest should say so; consumers pinning "the enhanced FRS" should be able to see its content date, not just a version string.
4. Certification surfaces need coverage floors. A dataset should not become the certified default by winning on a surface that omits a major benefit; rotation wins are only as meaningful as the surface is complete.
5. Ratios validating is not levels validating. Per-household agreement was read as health; the paper now states the aggregate layer separately for exactly this reason.

Measurement scripts and the full diagnosis trail: #452 (both diagnosis comments), PolicyEngine/microcosm#701.

[MaxGhenis]

Follow-up: origins of the remaining defects referenced above, traced on the tag lineage (blameless, same as the main writeup — each change was reasonable in context; the lesson column is what matters).

defect	born in	date	status
Bounded anti-convergent loss (torch.min in sre) — #456 defect 1	#161 (commit 72eca84)	2025-08-05	open; #456 proposes the log-ratio fix
OBR UC caseload emission dropped (dispatch carried as dead code)	#269	2026-02-15	fixed in #458 (dead dispatch deleted; explicit dwp/uc/households added)
Inert undercount_relative (×1.0002 instead of ~×1.21)	#269	2026-02-15	fixed in #458
Dwellings matched to household counts (ons_tenure) — #457	#269 (module created there)	2026-02-15	open (#457)
VOA chargeable-dwelling totals as household counts — #457	#269 branch (incl. review-fix commit 07e3452, which notably restored other dropped targets)	2026-02-16	open (#457)
Silent future-year target drop (closest > year → None, no log) — #456 defect 2	commit 23911bc ("correct student loan imputation year and drop future-year fallback")	2026-02-20	open; currently silently drops all 15 two-child-limit targets
Version-outruns-artifact pattern (2024-25 line switch leaving enhanced_frs_2023_24.h5 frozen under advancing version numbers)	#410	2026-05-23	design debt; see lesson 3
History cut (pre-May archaeology requires the tag lineage)	#398	2026-05-09	note for future spelunkers

Outside this repo, completing the picture: the policyengine.py certification PRs (policyengine.py#409/#427, June) adopted populace_uk_2023 as the UK certified default off a surface with no benefit targets (PolicyEngine/microcosm#701), and the August deductions validation (policyengine-uk#1815) pinned the February-era artifact when the healed April build already existed — the working paper's "~40% low" caveat measured that stale pin. Both of those were our own agent lanes.

The pattern worth naming: February's defects came from hand-written infrastructure sprints, spring's from agent-authored velocity, and none were caught because review examines the diff while these failures lived in the absence — an emission that stopped, a fallback that vanished, a surface that never had the family. Absence needs machine checks (coverage floors, output benchmarks), not sharper eyes.

[MaxGhenis]

Action taken on the certified-default consequence (layer 3 / lesson 4): policyengine.py#491 now reverts the UK certified default from populace_uk_2023 to enhanced_frs_2024_25 @ 1.56.16 — the artifact this postmortem's measurements show at 6.35m UC benefit units / £75.3bn (vs the certified 3.26m / £36.9bn). Certified on a publisher compatibility claim appended to the 1.56.16 release manifest with the measurement evidence; the full policyengine.py suite (783 tests) is the arbiter and passes. populace_uk_2023 and the frozen 2023-24 names stay resolvable at pinned revisions via dataset overlays.

The re-flip condition is written into the PR as a proposed standing rule: populace_uk returns as certified default only when it passes a benefits-inclusive surface with UC caseload and spend within 10% of the OBR/DWP counterparts — the coverage floor from lesson 4, made mechanical.