Skip to content

Baseline artifact pipeline phase 3: deploy consumes the store (image fetch, deployed markers, prewarm retired) #652

Description

@anth-volk

Phase 3 of the baseline artifact pipeline (phase 1: #643, phase 2: #648).

Phases 1–2 fill and verify a content-addressed GCS artifact store on every deploy; nothing consumes it yet. Phase 3 is the cutover:

  • Replace the in-image dataset prebuild layer (prebuild_country_datasets) with a self-contained fetch_artifacts layer that downloads every manifest-listed artifact (3 datasets + 60 cohort baselines, ~430 MB) from the store into POLICYENGINE_DATA_FOLDER, with a version freshness gate and loud failure on any missing object.
  • The deploy job consumes the precompute job's manifest_digest output: app.py resolves the manifest from GCS on the runner and passes its content as the fetch layer's args, so the Modal layer cache busts exactly when the artifact set changes.
  • Each deploy leg writes a deployed/<environment>.json marker after the health check — the liveness signal for the phase-4 GC.
  • Retire the manual prewarm ritual: delete prewarm_app.py, the prebuild layer, and the force_build escape hatch; migrate the coupling tests.

The merge is atomic by design: the producer already flows from phase 2, so this PR only adds the consumer.

Phase gate (post-merge): deploy green on both legs with the image build fetching in seconds instead of building; a beta economy request logs baseline_artifact=hit with p50 down; markers visible in the bucket for beta and prod; a fault drill proving delete-then-rerun healing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions