Run the backfill simulation on Modal; the workflow becomes trigger/commit only

[DTrim99]

Fixes the reform-validation backfill for good. #107's chunking was necessary but not sufficient: every scheduled run through 7/27 still dies — the logs show even a single 16-spec levels chunk receiving the runner shutdown signal ~5.5 minutes in, before its first simulation completes. One Microsimulation plus the populace/torch import stack no longer fits ubuntu-latest's 7GB, so no chunk size saves the hosted runner.

Design

The simulation moves to Modal; GitHub Actions keeps the schedule, the override-exists guard, and the commit/auto-merge PR role. Every tick now finishes in seconds:

• tick N — release lacks an override, Volume has no artifact → modal deploy + spawn the run (fire-and-forget, pinned to populace main's current sha for provenance) and exit. A started marker in the Modal workdir prevents double-spawns.
• tick N+1 — artifact on the Volume → download, gen-reform-overrides.mjs, lint, auto-merging override PR (unchanged from today's flow).

tools/reform_validation/modal_backfill_app.py runs the identical producer path — backfill.py mounted verbatim into a 64GB container — cloning populace at the requested ref and installing the release-exact policyengine-us/-core from the release manifest at runtime (engine pins vary per release, so they don't belong in the image). The workdir lives on the cd-reform-validation Volume, so batch partials survive any interruption and a re-spawn resumes rather than restarts.

This pattern has now produced three correct artifacts (Build O twice — including the post-populace#513 236-row payload on #109 — with the chunked-levels path exercised end-to-end).

What admins need to do

Add repository secrets MODAL_TOKEN_ID / MODAL_TOKEN_SECRET (PolicyEngine Modal workspace). Until they exist the workflow fails with an explicit error naming them. Compute cost is ~$1–2 per release; ticks with nothing to do never touch Modal compute.

Deployed + pre-seeded

The app is already deployed (cd-reform-validation-backfill) and the Volume is pre-seeded with the Build O artifact — so the first live tick after secrets land will harvest Build O directly if #109 hasn't merged by then (making #109 closable either way).

🤖 Generated with Claude Code

[vercel]

@DTrim99 is attempting to deploy a commit to the PolicyEngine Team on Vercel.

A member of the Team first needs to authorize it.

[PavelMakarchuk]

The design is right — the OOM evidence justifies moving off the hosted runner, the spawn-then-harvest two-tick split is clean, and I verified the resume claim is real (backfill.py checkpoints per-batch partials in the workdir, which now lives on the Volume). YAML parses, the app compiles, and the workflow and app derive the same rv_<sha8> workdir and artifact names. One blocking issue and two nits:

Blocking: a permanently failed run stalls the release silently, forever. The app writes the started marker before running the producer and never removes it. If backfill.py fails after that point, every subsequent tick sees rv_<sha8>/started, logs "already in flight," and exits 0 — the release never gets an override, nothing retries, and no workflow ever goes red. Since the workdir checkpoints make a re-spawn resume rather than restart, the fix is small: treat a marker older than some threshold (12–24h, comfortably past the 8h function timeout) as stale and re-spawn, or store the Modal call ID in the marker and check the call's status before deciding to wait. (Failures before the marker write — e.g. clone errors — already retry naturally, which is good; it's only post-marker failures that black-hole.)

Nit 1: the producer ref comes from an unauthenticated curl https://api.github.com/... — shared-runner IPs routinely exhaust the 60/hr anonymous rate limit, and the error JSON would crash the ['sha'] lookup. gh api repos/PolicyEngine/populace/commits/main --jq .sha with the built-in GH_TOKEN removes the flake.

Nit 2: harvested artifacts and workdirs accumulate on the Volume indefinitely. Trivial cost, fine to punt, but worth an eventual cleanup line in the harvest path.

Everything else held up: secrets validated with an explicit error naming them, quoted heredoc avoids shell expansion into the Python snippet, marker-check pipeline is safe under set -euo pipefail, triggers are schedule/dispatch only (no fork secret exposure), and installing release-exact engines at runtime rather than baking them into the image is the correct call. Happy to approve once the stale-marker handling and the gh api swap are in.

[DTrim99]

All three addressed in 128e0d6:

Blocking (stale marker): went with your stronger option — the spawner now records the Modal call id on the Volume (rv_<sha8>/call_id) and each tick checks the call's actual status via FunctionCall.from_id().get(timeout=0) before deciding: still running → wait; failed/vanished/finished-without-artifact → re-spawn (which resumes from the workdir checkpoints). The 24h age threshold survives as a fallback for markers with no readable call id (the started marker is now timestamped). Logic lives in tools/reform_validation/spawn_or_wait.py rather than the YAML heredoc. Verified all four branches against the live Volume: no marker → SPAWN, fresh marker/no call id → WAIT, 30h marker → SPAWN, dead call id → SPAWN (NotFoundError caught).

Nit 1: producer ref now comes from gh api repos/PolicyEngine/populace/commits/main --jq .sha with the built-in GH_TOKEN.

Nit 2: took it now rather than punting — the harvest path deletes the bulky rv_<sha8> workdir (H5 + partials) after a successful artifact download. The artifact json itself stays on the Volume deliberately: it's the done-signal that stops re-spawns until the override PR merges.

[DTrim99]

On hold (per David) — leaving open, not merging yet: the External checks surface this feeds is being retired in favor of policyengine-scorecard (#140, removal draft #141; history ingested in PolicyEngine/policyengine-scorecard#16). The Modal app + spawn_or_wait liveness machinery here (review fixes in 128e0d6) is the natural per-release producer to re-home in the scorecard once the automation question in PolicyEngine/policyengine-scorecard#15 is settled — at which point this closes in favor of that port, with your review carrying over.