RFC: What should the data-generation pipeline be — shaped by its consumers and validation?

[PavelMakarchuk]

Why this discussion

Over the last week several releases (buildi/buildj = sparse-rmloss100 calibration sweeps, buildl = acs-local multispine) shipped to production latest without staging telemetry, without reform_validation.json, and without a Slack alert. Each was patched downstream (consumer-side reform-validation backfill, bypass-proof alert in #407, a dashboard redeploy). That's three symptoms of one structural cause, so before adding more patches I want to agree on what the pipeline should be — designed backwards from what consumes the data and how we validate it.

Start from the consumers

The pipeline exists to serve four consumers. Each implies a specific validation — validation isn't a generic checklist, it's "what would break this consumer if the data were wrong."

Consumer	Needs	Failure mode if wrong
PolicyEngine-US microsim (primary)	H5 weights/imputations correct + stable	every downstream score wrong
Reform scoring / analyze-policy	dataset reproduces known fiscal scores (JCT/JCX)	reforms mis-scored, silently
calibration-diagnostics dashboard	a complete contract (all JSONs) per release	"bare release" holes on the dashboard
Staging reviewers	candidate-vs-production diff before promotion	regressions ship blind

Validation is already defense-in-depth — it just isn't enforced

• Build gates (gate_summary) — cheap/fast — microsim sanity (convergence, PUMA ladder, mass conservation)
• Staging compare (target-by-target diff) — medium — regressions vs current production (Stage US builds by default; record staging in the build manifest #270 made this default-on)
• Reform validation (in + out-of-sample vs JCT) — slow — calibration drift gates can't see
• External benchmarks — reality drift

The stack is good. The problem is that none of these gate promotion.

The one structural flaw

latest.json is an ungated mutable pointer, the safeguards are opt-out (--no-staging, skip-validation), and experiments share the production channel.

So the easy path is the unsafe path: a calibration sweep or a geography experiment can be built bare and promoted to latest with a manual pointer write, and nothing refuses it. The microsim, reform scoring, and dashboard all immediately consume it. We then patch symptoms downstream instead of closing the hole.

Proposal: invert the default

Experiments should be the cheap default; production should be opt-in and gated. You should have to earn latest, and you should be able to run a hundred sweeps a week without thinking about safeguards because they can't reach production anyway.

 EXPERIMENTAL          →   CANDIDATE (staging)      →   PRODUCTION (latest)
 sweeps, geography,        full validation runs         only a candidate that
 acs-local                 (gates + reform_val +        PASSED every gate can
 runs/ namespace           staging compare)             be promoted
 never `latest`            = a "release candidate"      promotion is GATED

Phase 1 — the promotion gate (keystone, do first)

Make latest.json movable only through publish_release --promote, which refuses unless the release satisfies the full contract: gates pass + reform_validation.json complete (out_of_sample_simulated=true) + staging telemetry present. We already have this instinct — publish_cli refuses incomplete reform-validation today. Generalize that one refusal into "the contract is the only door to latest." Small change to code we own; ends the bare-release class immediately.

Phase 2 — make experiments cheap and separate

Builder defaults output to a runs/ namespace (or the staging repo) that never touches latest. Sweeps/geography land there bare by design. Production requires an explicit --promote → triggers the Phase 1 gate. Now the default is safe and experimentation is unconstrained. This is directly adjacent to #398 (release contract needs dataset-role classes — national-default vs local-area artifacts).

Phase 3 — reform-validation as a candidate-phase gate (only when it hurts)

Move the slow out-of-sample validation into the candidate phase, run async, and have --promote wait on it. This retires the consumer-side backfill net. Don't build until Phases 1–2 are stable.

What I would not build

Experimental builds as first-class tracked artifacts with their own dashboard view — most machinery, least risk reduction. Revisit only if the research/geography workflow grows into a product.

The backbone: a versioned contract

One declared, schema-versioned set of {H5 + JSON files} that every promoted release must satisfy, with contents derived from consumers:

• microsim correctness → gates + calibration_diagnostics + reform-validation reproduces known scores
• reform scoring → out-of-sample validation + external benchmarks
• dashboard → contract completeness
• reproducibility → record the inputs: base-H5 sha, Ledger-facts hash (currently unrecorded), code sha, calibration recipe. A "blessed" release that can't be rebuilt isn't blessed.

The dashboard consumes the contract; the microsim consumes the H5; the gate enforces the contract once, at the door.

Related work

• Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398 — release contract dataset-role classes (national-default vs local-area) — Phase 2 depends on this
• ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403 — ACS-multispine staging probe
• Stage US builds by default; record staging in the build manifest #270 — stage US builds by default + record staging in the manifest
• Fire the release Slack alert from publish_release; warn when the webhook is unset #407 — release Slack alert made bypass-proof (fires from publish_release)
• calibration-diagnostics — the dashboard consumer + the scheduled reform-validation backfill automation (the current safety net)

Questions for the team

1. Do we agree latest should be reachable only through a contract-checking gate (Phase 1)?
2. Should the builder default experimental output to a non-latest namespace (invert the polarity), or keep explicit opt-out?
3. Is reform-validation a hard promotion gate, or does the consumer-side backfill stay the accepted safety net long-term?
4. What is the minimal versioned contract every promoted release must carry — and do we block promotion on the Ledger-facts hash for reproducibility?

---

Filed to frame the design before adding more per-symptom patches. Happy to turn Phase 1 into a PR against the standard flow if there's agreement.

[MaxGhenis]

Builder-side context, as requested — answers to your four questions plus two factual refinements. Overall: the diagnosis is right, Phase 1 is the correct keystone, and today's build day supplied fresh evidence for it.

One correction to the incident list: buildl (acs-local) never touched latest — it was published with the pointer explicitly untouched and verified afterward (that's what #397's --no-latest plus a direct-API publish were for). Its failure class is different and it's the one your Phase 2 depends on: the release namespace has no dataset-role classes, so a local-area artifact carrying a nonstandard diagnostics schema landed next to certified releases and the dashboard rendered it as "0 targets" instead of "different class / incompatible diagnostics." That's #398 surfacing in the UI. (A fix for the dashboard side — render any release whose diagnostics carry target rows, and show an explicit incompatible-diagnostics state instead of a silent zero — is in flight today.) buildi/buildj are fair as charged: the pointer moved on manual writes with validation opt-outable.

Q1 — gate the pointer? Yes, unambiguously. Two pieces of evidence from this weekend that enforcement-at-the-door works when it exists: (a) the #406 staged builder's code-identity lock refused to resume a checkpoint chain across a code change twice today — mildly annoying, correct both times; (b) the child-support signal gate stopped the first full-scale Build M base this afternoon on a 0.0047-vs-0.005 share and the investigation it forced surfaced a real modeling question (#417: the retired eCPS had silently drifted payer prevalence 3–4.5× above its survey source). Gates that block produce investigations; telemetry that observes produces backfills. And note the counter-evidence for the current design: I bypassed the whole publish driver with a direct API write for buildl — legitimately, for a non-default artifact — which proves the door is open to anyone. --promote as the only writer of latest.json closes it.

Q2 — invert the default namespace? Yes, and the infrastructure now exists: policyengine/populace-us-staging was created today (private, team intermediates; the Build J base lives there for your Build K). Natural mapping: experiments → staging repo by default; candidates → main repo releases/ with the pointer untouched; production → --promote only. One requirement from the buildl experience: the experimental default must still write sha256sums.txt + a minimal provenance manifest — cheap, and it's the difference between "experiment" and "mystery file" when someone picks it up a week later (your reproducibility point, applied to the cheap tier too).

Q3 — reform-validation as a hard gate? As a candidate-phase gate, yes — with the probe set versioned inside the contract, so "complete" is falsifiable. The reform-validation battery only means something relative to a declared probe list (which reforms, which oracle scores, in/out-of-sample split). If the contract pins reform_validation.probe_set_version, then the backfill net can retire without a definitional hole. Until then the backfill stays, explicitly labeled as the interim net.

Q4 — minimal contract + ledger hash? Refinement: the ledger linkage is more recorded than "currently unrecorded" — since the parity stack (#404/#405), the certified contract carries source_contract.ledger_commit, the fiscal-facts feed sha (the parity manifest itself pins 94b7155f… and assert_target_parity_manifest_current refuses drift), target_registry version, and the target-surface sha. What's missing is exactly your point applied one level up: --promote should verify those recorded values against the artifact being promoted, not trust the manifest. Minimal promoted-contract file set, from having assembled one by hand: release_manifest.json (per-file sha256s), build_manifest (code sha, base-H5 sha, seed, thread env), calibration_diagnostics, gate_summary, target_parity report, reform_validation (with probe-set version), staging-compare record, us_source_coverage, plus role-specific extras (reviewed_null_fills, held_back_columns for the local class). Everything on that list already exists for at least one release; the work is declaring it as one versioned schema and checking it at the single door.

Sequencing endorsement: Phase 1 → #398 role classes → Phase 2, and hold Phase 3 until the probe-set versioning lands. Also +1 to naming — happy to bikepaint post-launch; the build letters were checkpoint labels that escaped into production vocabulary, which is itself an argument for your pipeline.

🤖 Generated with Claude Code