UK wave 1: ONS mid-2023 population estimate source packages, UK countries + England regions (#132)

[juaristi22]

Part of #132 (wave 1 also covers the SLC packages, blocked on #131/#136, and the ISC private-school package — this PR does not close the issue).

Lane: ledger-source-ingestor (.github/ledger-agents.yml) — judges ledger-source-fidelity + ledger-boundary. Paths: packages/** plus the db/data/** writes the fetch-artifact harness flow prescribes (note for maintainers: the ingestor lane's allowed_paths doesn't list db/data/** today).

What

Two source packages backing the 108 region×age rows of the June production calibration surface, 1,196 facts total, at publisher granularity (single year of age; the fixture's 10-year bands are a populace-side mapping concern):

package	artifact	rows	facts
ons-mye-2023-uk-countries	mye23tablesuk.xlsx (MYE2 persons)	UK, Wales, Scotland, Northern Ireland	368
ons-mye-2023-england-regions	mye23tablesew.xlsx (MYE2 persons, 2023 LA-boundaries edition)	publisher-emitted E12000001–9 region rows	828

Shape: one record set per single year of age (groupby_dimension: age; the all-ages sets are age-totals), rows carry per-row geography overrides, and age enters fact identity via row filters mirrored verbatim by explicit constraints (the consumer-contract comparator matches unit/label exactly). The YAMLs are machine-generated and uniform (10.9k + 21.4k lines); if a compact templated form is preferred, that belongs with the contract follow-up below.

Sources (for review)

• UK countries workbook: dataset page, mid-2023 edition, mye23tablesuk.xlsx — sha256 7ee5724ecf71abc3d811c89174cdddf1bce7f07d3eefc8b69b15ee8371414b16
• England & Wales workbook: dataset page, mid-2023 (2023 LA boundaries) edition, mye23tablesew.xlsx — sha256 96dcbb73d01c2965a0e8173bdf736f04a16d91dc189c4a52220919792e5e2c94

Both artifacts were fetched twice by independent code paths (manual download, then uv run ledger fetch-artifact) with byte-identical checksums; provenance_class: census (census-anchored official estimates, mirroring the census_pep precedent).

Source decision this PR asks reviewers to ratify

Issue #132 originally framed this family as projections (SNPP/NPP), inheriting policyengine-uk-data's claimed lineage for its committed demographics.csv. Kickoff findings changed that:

1. The CSV's provenance is unrecoverable: last updated 2025-06-10, which predates the England 2022-based SNPP publication (2025-06-24); no vintage is recorded anywhere. No projection package can be shown to reproduce the production fixture either.
2. The production surface calibrates period 2023 only, and official actuals for mid-2023 exist — the ONS mid-year estimates used here (observation-class, census-anchored).
3. Deltas vs the production fixture, after resolving that uk-data's _age_80_89 band is actually 80+ (a systematic −18% otherwise): median 0.76%, p90 2.14%, max 2.94% — the expected estimates-vs-projections gap, enumerated per-target in the populace consumption gate (Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate populace#622). Top deltas: south_east_age_10_19 +2.94%, east_age_10_19 +2.86%, west_midlands_age_10_19 +2.46%.

Projection packages (2022-based NPP uk.zip, England 2022-based SNPP) remain queued for future-year calibrations, where actuals cannot exist — porting both sources is consistent with the exhaustive-vintage policy; populace picks the active source per calibration year.

Gates

• validate-package: clean for both packages, zero warnings.
• build-suite --year 2023: green end to end — source cells, source records, facts, DB build, agent_acceptance.json valid with zero errors (only the standard non-fatal no_concept_alignments warning; Axiom alignment is a later step).
• Facts spot-verified against the workbooks by an independent openpyxl read (UK all-ages 68,265,209; Wales age-0 27,701; Scotland 90+ 45,547; London age-35 151,827; North East all-ages 2,711,380 — all exact).
• Lane test files (test_source_files.py, test_ledger_source_cells.py, test_ledger_facts_only.py) pass.

Draft because: the consumer-contract export requires storage.r2 (raw_r2_uri) in the manifests. The committed manifests honestly record checksums and retrieval metadata only; the suites were proven green under a locally injected, disclosed, and reverted R2 block. After source review, a credentialed maintainer runs uv run ledger publish-raw --root db/data and the last gate closes with no package changes.

Contract findings for maintainers

1. Digit-text column headers are unguardable: _render_value int-coerces digit-string expectations while _resolve_guard_cell compares raw — MYE2's text-typed 0..89 headers can't be expressed as expected_column_header. The age span is pinned by the guarded non-digit endpoints (All ages, 90+) plus the sha-pinned artifact. Follow-up contract-lane issue to be filed.
2. ledger-agents.yml's ingestor lane omits db/data/** although the canonical fetch-artifact flow writes there.

🤖 Generated with Claude Code

[juaristi22]

Source-review outcomes and CI triage (commit ab98578).

Source decisions (reviewed):

• Wave-1 region×age ships the mid-year estimates (MYE). The _age_80_89 fixture band is proven to be 80+: the nine bands in policyengine-uk-data's demographics.csv sum to the all-ages totals at ratios 0.999–1.003, so the bands are exhaustive.
• National projections are now in scope and in this PR: the demographics.csv lineage problem only ever concerned the regional family. ons-uk-population-projections-2024 packages the 2024-based NPP principal (uk_ppp from the z1 uk.zip, sha dea79703…48ab1) — one record set per year 2024–2029, rows = sex × single year of age, 1,284 source_projection facts, spot-verified byte-exact. Later projection years land via the vintage backfill lane; regional projections stay deferred until a calibration year needs them.

CI triage: the red check was not the R2/storage gate — it was the SOURCE_PACKAGE_ALIASES drift guard (unregistered packages/* fail loudly rather than being silently dropped from bundles, per #78). All four packages are now registered, and the year-2023 bundle expectations are updated to the post-publication truth measured by a full local bundle build: fact_count 39,219 → 42,895, source_package_count 60 → 64, period_count 16 → 19, ons by-source 1,246 → 4,922, source tables 55 → 59, geography_count 1,053 → 1,065.

Remaining red, by design: the bundle test's consumer export requires raw_r2_uri, and the committed manifests honestly carry no storage.r2 until the artifacts are uploaded. One maintainer command closes it with no further code changes:

uv run ledger publish-raw --root db/data

(Counts were verified under locally injected and reverted storage.r2 blocks, disclosed as in the PR description; committed manifests remain upload-truthful. Four artifacts to publish: mye23tablesuk.xlsx, mye23tablesew.xlsx, mye24tablesuk.xlsx, uk.zip.)

🤖 Generated with Claude Code