Skip to content

Artifact pipeline phase 2: precompute job in the deploy pipeline (producer only) #648

Description

@anth-volk

Phase 2 of the baseline artifact pipeline (phase 1: #643, shipped in #644). Make the artifact precompute a first-class job in the deploy pipeline, with nothing consuming its output yet — the image build and runtime stay exactly as they are today.

Scope:

  • A modal-precompute.sh runner script wrapping the phase-1 manual invocation (uv run modal run src/modal/precompute_app.py), parsing the app's MANIFEST_DIGEST= stdout contract into a manifest_digest GitHub Actions output and failing loudly when it is absent.
  • A precompute job in the reusable deploy workflow ahead of deploy, on both legs: syncs Modal secrets itself (no dependence on a prior deploy's sync), runs the app against the leg's Modal environment with the repo-level POLICYENGINE_ARTIFACT_BUCKET variable, exports the manifest digest as a job output. deploy gains needs: [precompute] and nothing else — the digest stays deliberately unconsumed until the phase-3 image cutover.
  • A force_recompute workflow_dispatch input threaded to the script, mirroring force_latest. Write-once still holds: a forced run recomputes but never replaces an existing store object.

Consequences: ordinary deploys pay a fast no-op (store already warm, ~minutes); a pin-bump deploy pays the full dataset+baseline recompute inline (~15–20 min measured in phase 1), replacing the manual prewarm ritual with a visible, measured pipeline step.

Phase 2 gate after merge: precompute job green on both legs of a main deploy; the prod leg a measurable no-op (the store is global — beta warms it); deploy + integ behaviorally unchanged and green.

Later phases (separate issues): image fetch layer replacing prebuild_country_datasets (retiring the prewarm ritual and force_build); store/app GC; first measured pin-bump deploy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions