Skip to content

Artifact pipeline phase 1: content-addressed keys, store, runtime reader, precompute app #643

Description

@anth-volk

Part of the baseline artifact pipeline: precompute the single-year US datasets (2026, 2027, 2025) and the 20 per-cohort national baselines per year as part of build/deploy, cache them content-addressed in GCS, bake them into the Modal image, and have the runtime load baselines instead of computing them.

Under the segmented-national default (#637), every plain US national macro request fans out to 20 fixed state-cohort children, and each child's baseline half is identical across all default-baseline requests — recomputed every time. Separately, the dataset prebuild layer's cost is paid through a manual prewarm ritual before pin-bump merges.

Phase 1 (this issue):

  • artifact_keys: canonical-JSON sha256 keys digesting the full input closure (versions, receipt content sha, certification fingerprint); deterministic Simulation ids; store layout. Keys carry country/region/scope so UK or state-level baselines need no schema change later.
  • artifact_store: GCS client with write-once uploads, content-addressed manifests, deployed-environment markers.
  • Runtime reader: qualifying baselines get deterministic ids; ensure() loads a baked artifact when present; loads are validated column-by-column against resolve_entity_variables and fall back to run() on any gap. No artifacts present -> today's behavior exactly.
  • precompute_app: ephemeral modal run app that plans against the store, computes only misses (wave 1 datasets, wave 2 cohort baselines via the executor's own request path), runs a determinism gate, and publishes the deploy manifest.

Later phases (separate issues): CI precompute job in the deploy pipeline; image fetch layer replacing prebuild_country_datasets (retiring the prewarm ritual and force_build); store/app GC; first measured pin-bump deploy.

Phase 1 gate before merge: full suite green; manual staging run of the precompute app populating the store (needs the artifact bucket provisioned + POLICYENGINE_ARTIFACT_BUCKET set), with a measurable no-op on immediate re-run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions