Skip to content

NZ source conventions: R2 country layout, tax-year labelling, geography rulings #175

Description

@juaristi22

Lane: ledger-contract-maintainer (.github/chronicle-agents.yml) — judges ledger-contract + ledger-boundary
Blocks: #176 (all credentialed R2 publishes; the IRD batch for the tax-year note; the MSD/MBIE batches for the geography rulings). The stats_nz batches can be authored in parallel; their R2 commit waits on this issue.

Context

New Zealand onboarding (epic microcosm#343) is the third country entering chronicle. Verified against the current vocabularies, no closed-set changes are needed — NZ rides existing levels, periods, parsers, and free-form units (nzd). What is needed is one storage-layout precondition, one comment-level contract note, and four recorded rulings, isolated here so no source-package PR drags through contract review (the #131 pattern).

1. Precondition: country-organized R2 layout

Today ledger-raw keys are raw/<source_id>/<package_id>/<year>/<sha256>/<filename>, recorded in each db/data/**/manifest.yaml under storage.r2.{key,uri}. Before any NZ bytes are published:

  • New convention: a country segment after the prefix — raw/nz/<source_id>/... and raw/uk/<source_id>/.... ledger-derived follows the same convention for new uploads.
  • US keys stay at the legacy raw/<source_id>/... untouched — deliberately deferred to avoid breaking downstream consumers of existing URIs (Supabase mirror rows, built consumer artifacts pinning source.raw_r2_uri, populace-side fetches). Migrating the US is a separate future issue once its consumers are enumerated.
  • UK migration: server-side copy (not move) of every UK object to its raw/uk/... key — keys are content-addressed, so the sha256 segment is unchanged and byte-identity is verifiable from the key itself. Then one manifests-only sweep PR rewriting storage.r2.{key,uri} across all UK db/data/**/manifest.yaml (the credentialed-commit class, cf. "Record R2 storage for wave-1 ONS artifacts after credentialed publish-raw"; cross-cutting sweep shape, cf. Require provenance_class (measurement basis) on every record set and fact #96). Verification: a full bundle build with agent_acceptance.json's raw_artifacts_have_r2 green for every UK package. Old UK keys are deleted only in a later follow-up after a soak period, not in this PR.
  • Tooling: chronicle fetch-artifact / publish-raw already accept --r2-prefix (default raw). This issue adjudicates: either prescribe --r2-prefix raw/nz / raw/uk operationally in issue text, or add a country-aware default derived from the package's publisher directory. Recommendation: the country-aware default, so a forgotten flag cannot scatter objects across schemes; publish-raw should also refuse a manifest whose recorded key disagrees with the derived country prefix.
  • No new bucket; wrangler login auth unchanged; the raw/<sha256>/ content-addressing within the key is unchanged.

2. Tax-year labelling note (chronicle/core.py comment block)

NZ labels its April–March tax year by the ending year: publisher "2024 tax year" = 1 Apr 2023 – 31 Mar 2024. Under the single-year-label rule this is stored as tax_year: 2024 — the opposite of the UK split-label opening-year convention (FY2024-25 → 2024). One sentence in the period comment block naming the NZ case, so no future author "corrects" NZ periods to opening years. Every NZ tax-year record set must carry PeriodCoverage start/end dates and source_period_label.

3. Recorded rulings (no code change)

  • Territorial authorities ride the existing local_authority level with geography_vintage: ta_2025 — the vintage disambiguates the code system exactly as lad_2023 vs lgd_2014 does for the UK. No new level.
  • MSD Work & Income regions (the only regional Accommodation Supplement grain published) ride statistical_scope with geography_vintage: msd_wi_region and stable slug ids (nz-wi-…) — direct precedent: Scottish BRMAs at statistical_scope/brma_scotland in ons/pipr_rents_by_area_june_2026.
  • SA2 is deferred to wave 3 (NZ source packages, wave 3: local geography (TA / SA2) #178). When needed it merits a first-class sa2 level (a general statistical geography like msoa), not statistical_scope.
  • Rent quartiles are inexpressible today (ALLOWED_AGGREGATIONS has no percentile). Wave 1 ships median rent (in-vocab) and the geometric mean as aggregation: mean with concept_relation: approximate and evidence notes naming it geometric. A parameterized percentile aggregation is a future deliberate core change — do not smuggle quartiles through median.

Acceptance

  • UK objects exist under raw/uk/...; every UK manifest records the new key; full bundle build green including raw_artifacts_have_r2; old keys still present (deletion deferred).
  • The tooling decision is implemented (or documented as operational flags) and stated in NZ source packages, wave 1: the populace-nz v1 16-margin surface #176.
  • The core.py NZ tax-year sentence lands; docs/schemas/ untouched (no schema change).
  • The four rulings are recorded in this issue and referenced from docs/pe-nz-source-checklist.md.
  • Deterministic checks green: schema validation, consumer-contract validation, package import compatibility, raw-facts boundary validation. No existing package's behavior changes.

Out of scope

US key migration (future issue, consumers enumerated first); deletion of old UK keys (follow-up after soak); any NZ package content (#176).

Sizing

One contract-lane PR (tooling + comment + docs) + one credentialed manifests-sweep PR for the UK re-keying.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions