Skip to content

UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135

Description

@juaristi22

Lanes: ordinary maintainer tooling under db/; if the suite wiring touches policyengine_ledger/** consumer surfaces, that PR runs under ledger-contract-maintainer (.github/ledger-agents.yml)
Related: the academic_year period type is its own contract-lane issue (#131); PR 3 below waits on the wave-2 packages (#133)

Three small, independent enablers for the UK target migration.

PR 1 — de-hardcode db/pe_source_inventory.py

DEFAULT_PE_UK_ROOT = Path("/Users/maxghenis/PolicyEngine/policyengine-uk-data") (and the US twin) key the UK ingestion inventory to one person's checkout. Replace with env vars / CLI arguments with no default, and fail with a clear message when unset. The inventory's UK rows (ons 13, hmrc 6, dwp 5, obr 3, voa 2, nrs 2, nomis 2, slc 1, nts 1) are the working checklist for waves 1–3, so this lands before the waves start in earnest.

PR 2 — UK consumer-artifact suite

Add the UK suite wiring so ledger build-suite / build-consumer-artifact can produce a UK bundle covering the wave 1–3 packages + uk_firms + uk_local_geography + uk_national profiles. Known behavior to design around (observed on the US side, June 2026): a single---year bundle build drops year-pinned sources; the working pattern is per-suite build at each pinned year + concatenated consumer_facts.jsonl. Either fix the year handling properly here or codify the per-suite+concat pattern for the UK suite — pick one and document it.

PR 3 — retire the legacy hardcoded UK ETLs

db/etl_obr.py and db/etl_ons.py are hardcoded March-2024 dicts feeding the legacy three-table schema. docs/agent-source-package-harness.md already declares the old ETL loaders "compatibility and migration inputs, not the preferred agent" lane. Once the #133 OBR/ONS packages land: delete these two modules (or reduce them to explicit tombstones pointing at the packages), so no future agent mistakes them for the ingestion path. Verify the wave-2 packages supersede every value they carry before deletion.

Acceptance

  • Inventory runs against arbitrary checkout paths; CI proves the no-default failure mode.
  • One documented UK suite command produces a hash-pinned UK consumer artifact.
  • Legacy UK ETLs gone or tombstoned, with supersession verified in the PR description.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions