Skip to content

Resumable stage checkpoints for the pool builder: preemptions cost minutes, not builds (#578 inc 2) - #599

Merged
MaxGhenis merged 3 commits into
mainfrom
pool-stage-checkpoints-578
Aug 3, 2026
Merged

Resumable stage checkpoints for the pool builder: preemptions cost minutes, not builds (#578 inc 2)#599
MaxGhenis merged 3 commits into
mainfrom
pool-stage-checkpoints-578

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Cloud containers get preempted at 4–8h; the pool build needs 6–8h and its only resumable state was the primary-QRF directory — run 5 burned four attempts (~$67) with zero completions because every respawn restarted assembly, transfer, seed, and simulate from scratch.

This adds identity-guarded checkpoints at three boundaries, following the dense tool's checkpoint doctrine (correctness over cache warmth, ledgered materializer version):

Checkpoint Work preserved on resume
assembled (pre-clone) source loading, ACS mapping, assembly
transferred clone, 65 primary-QRF targets, tail transfer, ACS targets
simulated derive/seed and the 681-batch SSI evaluation

Identity binds all six input SHA pins + sizes, period, seed, engine version, operator registries/orders, estimator/batch settings, and the take-up contract, under a ledgered version constant. Invalid or corrupt checkpoints are ignored with named receipts and rebuilt; the deepest valid stage resumes; the agreement gate always runs fresh; the manifest records which stages were resumed with their identity digests.

Verification: 15 checkpoint/resume tests (round-trip byte-identity on canonical Frame serialization; identity-mismatch → rebuild; corruption → named failure + rebuild); #583 guard 495; input-surface guards 137 + 3; full workspace 4,952 passed / 59 skipped. Ruff + format + git diff --check clean.

Disclosed: production stage sizes/timings unmeasured locally (fixture-scale receipts); final PyTables artifacts are content-identical but not raw-byte-identical due to nondeterministic HDF metadata — canonical serialization is byte-identical; legacy unbound primary-QRF dirs are deliberately not auto-trusted.

Part of #578 increment 2. Run 6 dispatches on this merge with checkpoints on the volume.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Production stage-profile receipts, closing the disclosed "production sizes/timings unmeasured" gap: run 5 on Modal (16 cores / 330 GiB) wrote stage_profile.json to the volume before its container was preempted. All 65 primary-QRF stages succeeded in 30 minutes total; the phase's durable state is confirmed cheap to rebuild but the phases around it are not — attempt 4 died ~5h15m in, deep in derive/seed/simulate, exactly the work the simulated checkpoint preserves.

Top memory spikes (informs run-6 container sizing — 330 GiB is right, 256 would OOM):

Stage Wall Peak RSS
045_rental_income_would_be_qualified 142s 241 GB
043_farm_rent_income_would_be_qualified 150s 223 GB
062_first_home_mortgage_origination_year 85s 222 GB
041_estate_income_would_be_qualified 121s 216 GB
044_partnership_s_corp_income_would_be_qualified 122s 176 GB

Run 5 is now terminated by the no-attempt-5 rule (4 preemption deaths, zero completions, ~$122 total). Run 6 dispatches on this PR's merge, resuming the volume's primary-QRF directory plus these checkpoints.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant