[docs] Design sketch: content-addressed microdata publishing

[MaxGhenis]

Design sketch for what we'd build if publishing PolicyEngine microdata from scratch. Explicitly a sketch, not an accepted plan — written to capture the shape while today's session has the context loaded.

Motivation

Today's stack layers PyPI (country models), PyPI and HF (country data), GitHub repos (build code), and policyengine.py release manifests (sha256 pins). Refreshing any link touches six files across three repos and mixes identity ("what dataset is this?") with distribution ("where do I get it?") and discovery ("what's current?").

Two pain points from today's session that the sketch resolves:

1. HF is a model-repo or dataset-repo. us-data publishes as a model repo, uk-data is private-gated; Bump us-data 1.73.0 → 1.78.2 + fix HF model/dataset repo detection #310 hit this. A plain object-store path has one shape.
2. "Is our data out of date?" required spelunking because inputs.model_package.sha256 isn't stored anywhere queryable. Under the sketch, CI diffs it against the active channel's manifest.

Shape

s3://policyengine-data/
  us/{build_id}/
    enhanced_cps_2024.parquet     # content-addressed payload
    manifest.json                 # 2 KB: schema, inputs, digests
  channels/us/
    latest.json                   # { "build_id": "..." }
    stable.json

• build_id = sha256 of inputs (data vintage + model wheel + calibration sha + seed + lockfile). Deterministic; retagging impossible.
• Channels are tiny JSON pointers. Bumping stable is one S3 PUT.
• manifest.json carries schema + entity map, so agents learn column types without downloading 100 MB.
• pe.py's consumer resolves channel → build_id → manifest → payload with sha256 verification.

What stays

• Country data repos still own construction (boundary from release-bundles.md intact)
• TRACE TRO sidecars still ship with policyengine.py, same trov: / pe: vocabulary
• UK privacy boundary: bucket can be org-gated

Migration cost

~3 engineer-weeks if genuinely pursued.

Explicit non-goals

• Doesn't solve model drift (pe-us 1.653 and 1.700 will still produce different enhanced CPS)
• Doesn't solve calibration (still an open research problem)

Open questions (in the doc)

• GCS vs S3
• Parquet vs HDF5 for new builds
• Channel-history signing (transparency log)
• Single bucket or per-country namespaces

This PR

Adds docs/data-publishing-design.md and wires it into the Quarto nav under Reference. No code change. Merge-neutral for v4.x — this is a research doc.

[MaxGhenis]

Stress-test review folded in (commit 10fa058)

Subagent stress-tested the sketch against five scenarios. Four needed explicit design answers; one meta-flag affected framing.

Core architecture held up. The identity/distribution/discovery separation and the channels+manifest layer survive the review intact. No change to the shape.

What changed in the doc:

1. Honest framing up top. The motivating pains (Bump us-data 1.73.0 → 1.78.2 + fix HF model/dataset repo detection #310's HF repo-type bug and "is our data stale?") are fixable with a one-time HF migration + a 50-line CI job. The sketch is a cleaner architecture, but adopting it is a "where do we want to be in a year?" question, not an immediate-fix. Readers should pick it for architectural reasons, not because they think it solves those two bugs.

2. New "Unresolved risks" section covering all five scenarios with concrete "decision needed" notes:

   • UK Data Service audit trail (today HF logs downloader identity; sketch needs explicit gating on manifest fetches + resolver-hit logging)
   • Silent-promote attack (channel JSON is unsigned → sketch is strictly weaker than PyPI/HF until channel signing ships → must promote signing to v1)
   • Non-deterministic builds (today's Enhanced CPS pipeline has known non-determinism → need If-None-Match conditional writes or explicit first-writer-wins semantics)
   • Licence revocation vs immutability (tombstones + qualified replicability guarantee)
   • Cross-cloud replication (payloads mirror trivially; channels don't)

3. Revised cost estimate: 8–12 engineer-weeks, recategorised as v5 scope. The earlier "3 weeks" was ~3x optimistic.

The core three-concepts model is unchanged — just the operational realism around it.

[MaxGhenis]

Codex review folded in — structural rewrite (commit f951f5a)

Codex caught a load-bearing overclaim the first review missed: the sketch slid between recipe-addressed (sha256 of declared inputs) and content-addressed (sha256 of output bytes). Those are meaningfully different, and the earlier draft used "content-addressed" to claim properties only true of the latter.

Second, Codex flagged that the earlier framing ("replace PyPI + HF + GitHub + release manifests") was actively regressive: it proposed collapsing a scientific-citation-and-certification surface into a storage primitive. That's the pattern where a separate certification layer reappears on top within 2 years (codex: 60–85% probability), and we'd end up with two systems.

What changed in the rewrite

• Scoped to storage substrate only. release-bundles.md remains authoritative for certification + citation. The substrate is infrastructure underneath, not a replacement.
• Primary identifier is artifact_sha256 (sha256 of output bytes), not build_id (sha256 of inputs). Matches what OCI / Nix actually do in the parts that work.
• Dropped stable and lts-* channels. Codex: "stable" for microdata can mean four different things. Kept only latest (operational) and next (staging → feeds certification). "Authoritative" stays on the release-bundle side.
• Dropped the "org-independent identity" claim. data_vintage: "cps_asec_2024" is a label, not a hash; built_at/built_by break bitwise identity anyway. The current release-bundles schema actually records raw-input hashes, so regressing on that would be real.
• New section: "The release-bundle boundary (what doesn't change)" makes the preservation explicit.
• Honest cost table: 7–11 engineer-weeks, independent tracks, v5 scope at most.
• Honest "whether to pursue": keep the storage idea, drop the replace-release-bundles framing, don't build it to fix Bump us-data 1.73.0 → 1.78.2 + fix HF model/dataset repo detection #310 or "is data stale" (cheap targeted fixes exist), revisit if UK Data Service relationship gets stricter.

The doc's load-bearing sentence, now explicit: "When a release bundle is certified, it promotes an artifact from channels/next to a concrete artifact_sha256 pin in the country release manifest. After that, the release manifest is what papers cite; the storage channel is just the cache."

Both prior review's risks (UK audit, silent-promote, non-determinism, licence revocation, cross-cloud mirroring) carried forward. "Non-deterministic builds" is actually cleaner under output-hash identity — two runs produce two distinct artifact_sha256 values rather than silently colliding.