Chunk the baseline-levels batch so the reform-validation backfill survives the hosted runner - #107
Merged
PavelMakarchuk merged 1 commit intoJul 23, 2026
Conversation
The scheduled reform-validation backfill has been OOM-killed on every run
since 2026-07-21 ("runner has received a shutdown signal" ~4 min into the
levels batch), leaving Build M and Build O with no reform_validation.json.
The levels suite grew from 74 to ~176 specs when the Census SPM state-rate
backtests joined default_baseline_level_specs(), and scoring them all
against one accumulating baseline simulation exceeds ubuntu-latest memory.
Score levels in LEVELS_CHUNK=16 slices, each in its own short-lived
subprocess with a fresh baseline (same isolation the reform batches already
use); merge is unchanged (concat + dedup by id). Also install CPU torch
explicitly - the default index resolves ~4GB of CUDA wheels the runner
never uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@DTrim99 is attempting to deploy a commit to the PolicyEngine Team on Vercel. A member of the Team first needs to authorize it. |
DTrim99
added a commit
to DTrim99/calibration-diagnostics
that referenced
this pull request
Jul 23, 2026
…loss100-22bd902-20260722T232627Z (Modal reproduction) The scheduled workflow was OOM-broken until PolicyEngine#107 merged (its 18:26Z run today still ran pre-fix code), so this override was produced on a 64GB Modal container running the identical producer path (tools/reform_validation/backfill.py, unmodified) at the release-exact policyengine-us 1.764.6 / policyengine-core 3.26.11; provenance in the file's _backfill_note. 226 rows - the first payload carrying the Census state SPM poverty-rate suite (104 percent-unit rows, rendered by PolicyEngine#108). Producer predates populace#513, so the State reform suite is the original 8 rows. Since the scheduled workflow skips releases that already have an override, the expanded-registry regeneration for this release comes from a follow-up Modal run against post-#513 main (superseding commit to this file); Build P onward picks up the expanded registry automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DTrim99
added a commit
to DTrim99/calibration-diagnostics
that referenced
this pull request
Jul 27, 2026
…loss100-22bd902-20260722T232627Z (Modal reproduction) The scheduled workflow was OOM-broken until PolicyEngine#107 merged (its 18:26Z run today still ran pre-fix code), so this override was produced on a 64GB Modal container running the identical producer path (tools/reform_validation/backfill.py, unmodified) at the release-exact policyengine-us 1.764.6 / policyengine-core 3.26.11; provenance in the file's _backfill_note. 226 rows - the first payload carrying the Census state SPM poverty-rate suite (104 percent-unit rows, rendered by PolicyEngine#108). Producer predates populace#513, so the State reform suite is the original 8 rows. Since the scheduled workflow skips releases that already have an override, the expanded-registry regeneration for this release comes from a follow-up Modal run against post-#513 main (superseding commit to this file); Build P onward picks up the expanded registry automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DTrim99
added a commit
to DTrim99/calibration-diagnostics
that referenced
this pull request
Jul 27, 2026
Regenerated on Modal at the same release-exact engine versions with the producer at populace 51fbd3f (post-#513 registry expansion): adds the six tracker-informed state credit/deduction bills, three federal benchmark rows (ARPA CTC, ARPA childless EITC, CBO all-rates-+1pt), and the $1,200/person UBI mechanical check to the buildo override. Chunked-levels path from PolicyEngine#107 exercised end-to-end on this run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PavelMakarchuk
pushed a commit
that referenced
this pull request
Aug 3, 2026
* Backfill reform_validation.json for populace-us-2024-buildo-sparse-rmloss100-22bd902-20260722T232627Z (Modal reproduction) The scheduled workflow was OOM-broken until #107 merged (its 18:26Z run today still ran pre-fix code), so this override was produced on a 64GB Modal container running the identical producer path (tools/reform_validation/backfill.py, unmodified) at the release-exact policyengine-us 1.764.6 / policyengine-core 3.26.11; provenance in the file's _backfill_note. 226 rows - the first payload carrying the Census state SPM poverty-rate suite (104 percent-unit rows, rendered by #108). Producer predates populace#513, so the State reform suite is the original 8 rows. Since the scheduled workflow skips releases that already have an override, the expanded-registry regeneration for this release comes from a follow-up Modal run against post-#513 main (superseding commit to this file); Build P onward picks up the expanded registry automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Supersede with the post-populace#513 payload (236 rows) Regenerated on Modal at the same release-exact engine versions with the producer at populace 51fbd3f (post-#513 registry expansion): adds the six tracker-informed state credit/deduction bills, three federal benchmark rows (ARPA CTC, ARPA childless EITC, CBO all-rates-+1pt), and the $1,200/person UBI mechanical check to the buildo override. Chunked-levels path from #107 exercised end-to-end on this run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The scheduled reform-validation backfill (
reform-validation-backfill.yml) has failed on every run since 2026-07-21 — the runner receives a shutdown signal ~4 minutes into the first batch, which is the hosted-runner signature for OOM. As a result Build M and Build O reached the dashboard with noreform_validation.json(no override committed, nothing on HF).Root cause: the
levelsbatch scores everydefault_baseline_level_specs()row against one accumulating baseline simulation in a single subprocess. That suite grew from 74 specs (when the buildi/buildj backfills succeeded) to ~176 when the Census SPM state-poverty-rate backtests were added to the default set in populace — and the person-level state-rate computations push the process past ubuntu-latest's 7GB.The fix
LEVELS_CHUNK = 16specs per short-lived subprocess, each with a fresh baseline — the same isolation the reform batches already use (CHUNK = 2).mergeis unchanged (concat + dedup by id), so the merged artifact is identical to the monolithic payload.nvidia-*packages, visible in the failed-run logs) that the producer never uses and that eat the runner's disk/memory headroom.Verification
python -m py_compileon the patched script;--plan/--only/--mergemodes all flow through the same_batches()generator so partial names stay consistent within a run.After merge I'll dispatch the workflow against Build M to confirm the fix end-to-end and backfill the remaining gap.
🤖 Generated with Claude Code