You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cloud containers get preempted at 4–8h; the pool build needs 6–8h and its only resumable state was the primary-QRF directory — run 5 burned four attempts (~$67) with zero completions because every respawn restarted assembly, transfer, seed, and simulate from scratch.
This adds identity-guarded checkpoints at three boundaries, following the dense tool's checkpoint doctrine (correctness over cache warmth, ledgered materializer version):
Identity binds all six input SHA pins + sizes, period, seed, engine version, operator registries/orders, estimator/batch settings, and the take-up contract, under a ledgered version constant. Invalid or corrupt checkpoints are ignored with named receipts and rebuilt; the deepest valid stage resumes; the agreement gate always runs fresh; the manifest records which stages were resumed with their identity digests.
Disclosed: production stage sizes/timings unmeasured locally (fixture-scale receipts); final PyTables artifacts are content-identical but not raw-byte-identical due to nondeterministic HDF metadata — canonical serialization is byte-identical; legacy unbound primary-QRF dirs are deliberately not auto-trusted.
Part of #578 increment 2. Run 6 dispatches on this merge with checkpoints on the volume.
Production stage-profile receipts, closing the disclosed "production sizes/timings unmeasured" gap: run 5 on Modal (16 cores / 330 GiB) wrote stage_profile.json to the volume before its container was preempted. All 65 primary-QRF stages succeeded in 30 minutes total; the phase's durable state is confirmed cheap to rebuild but the phases around it are not — attempt 4 died ~5h15m in, deep in derive/seed/simulate, exactly the work the simulated checkpoint preserves.
Top memory spikes (informs run-6 container sizing — 330 GiB is right, 256 would OOM):
Stage
Wall
Peak RSS
045_rental_income_would_be_qualified
142s
241 GB
043_farm_rent_income_would_be_qualified
150s
223 GB
062_first_home_mortgage_origination_year
85s
222 GB
041_estate_income_would_be_qualified
121s
216 GB
044_partnership_s_corp_income_would_be_qualified
122s
176 GB
Run 5 is now terminated by the no-attempt-5 rule (4 preemption deaths, zero completions, ~$122 total). Run 6 dispatches on this PR's merge, resuming the volume's primary-QRF directory plus these checkpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloud containers get preempted at 4–8h; the pool build needs 6–8h and its only resumable state was the primary-QRF directory — run 5 burned four attempts (~$67) with zero completions because every respawn restarted assembly, transfer, seed, and simulate from scratch.
This adds identity-guarded checkpoints at three boundaries, following the dense tool's checkpoint doctrine (correctness over cache warmth, ledgered materializer version):
Identity binds all six input SHA pins + sizes, period, seed, engine version, operator registries/orders, estimator/batch settings, and the take-up contract, under a ledgered version constant. Invalid or corrupt checkpoints are ignored with named receipts and rebuilt; the deepest valid stage resumes; the agreement gate always runs fresh; the manifest records which stages were resumed with their identity digests.
Verification: 15 checkpoint/resume tests (round-trip byte-identity on canonical Frame serialization; identity-mismatch → rebuild; corruption → named failure + rebuild); #583 guard 495; input-surface guards 137 + 3; full workspace 4,952 passed / 59 skipped. Ruff + format +
git diff --checkclean.Disclosed: production stage sizes/timings unmeasured locally (fixture-scale receipts); final PyTables artifacts are content-identical but not raw-byte-identical due to nondeterministic HDF metadata — canonical serialization is byte-identical; legacy unbound primary-QRF dirs are deliberately not auto-trusted.
Part of #578 increment 2. Run 6 dispatches on this merge with checkpoints on the volume.
🤖 Generated with Claude Code