Per-target banking for the ACS transfer: preemptions mid-phase cost one target, not the phase (#578) - #608
Merged
Merged
Conversation
This was referenced Aug 4, 2026
MaxGhenis
added a commit
that referenced
this pull request
Aug 6, 2026
…-zero doctrine, by-origin battery (#578 revision pilot) (#616) * Stack ASEC + a seeded ACS household sample into one origin-labeled spine (#578 revision item 1) The ratified increment-2 revision removes the two-spine agreement seam: assemble_stacked_spine() draws a whole-household ACS sample (exact-count rule floor(fraction * eligible), seeded, deterministic over equal frames), reuses the reviewed assemble_spines seam unchanged, and freezes a stacked- spine manifest binding fraction, seed, the realized selection digest, and per-arm weight-harmonization receipts to the live rows. Origin labels survive as the ordinary support-channel columns. Weight harmonization follows the two-arm P-lineage precedent (each arm receives a declared share of the anchor mass; the seeded sample's inverse-fraction upweighting falls out of the mass allocation and is receipted, not assumed). validate_stacked_spine_frame() fails closed on any mutation of the sample, counts, digest, shares, or manifest. stacked_spine.py is registered as a source-spine provenance OWNER in the blindness guard (a reviewed contract change): stacking, gap-fill routing, activation authority, and the by-origin battery are origin-aware by charter, so population operators can stay spine-blind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Null-vs-zero doctrine, banked cross-origin gap-fill, and the single PUF pass (#578 revision items 2-3) The doctrine (audit item 1, binding): absence stays null until an authorized stage fills it. finalize_us_puf_tax_detail_predictions gains an absent_cells policy — the default reproduces the historical global fillna(0.0) byte for byte for the retiring lineage, while PUF_ABSENT_CELLS_PRESERVE_NULLS (hard-wired in every stacked entrypoint) creates missing columns as null, keeps coercion from manufacturing values, and scopes the donor-rate sparsification rewrites to the PUF clone arm so no boundary converts absence into an observed zero. The audited defect is pinned as a regression test in both directions. Strict recipient predictors (audit items 2-3): prepare_us_puf_tax_detail_chain_inputs / impute_us_puf_tax_detail_support gain require_complete_recipient_predictors — recipient features are built null-preserving and a missing predictor value on any recipient row is a named terminal failure instead of the silent zero-fill that collapsed ACS clone-1 draws to ~$2 degenerates in run 7. Gap-fill (item 2): gap_fill_stacked_spine() fills survey-specific fields cross-origin on the unclonled stacked spine under DECLARED activation authority — per direction, every target column must exist, donor-origin rows must observe it completely, and every null must lie on the declared recipient origin, verified before and after the fit (donor cells must be byte-identical). Donors are this owner's projection of the donor origin's native rows (authoritative rows, audit item 3); the spine-blind transfer_acs_inputs machinery and the #608 per-target banking store are reused unchanged, and the native ACS aggregates (acs_interest_dividend_rental_income et al.) serve as recipient predictors through the existing feature surface. Single PUF pass (item 3): clone_us_frame_for_puf_support gains a seeded whole-household clone attachment (floor(fraction * eligible), explicit seed, manifest-bound selection digest, pair-symmetric half weights, fraction=1.0 byte-identical to the full clone) and run_stacked_puf_pass() runs the primary QRF once over the stacked spine after gap-fill under both doctrines, with per-origin recipient receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pre-simulation completeness gate, by-origin battery, and the pilot end-to-end (#578 revision items 4-6) Completeness gate (item 4, audit item 5): stacked_completeness_gate() proves every declared entity/family/target is filled or carries an explicit AbsenceProof naming the origin channel and clone role whose cells may be null (with the reason receipted). A missing column and every unproven null are named terminal failures with per-origin, per-role counts — the check that would have caught run 7's silent 58-target skip, now paired with the preserve-nulls doctrine so a skipped family presents as nulls rather than observed zeros. By-origin battery (item 5): by_origin_battery() replaces spine-vs-spine agreement with the same statistics scoped to gap-filled families and per-family DECLARED metrics — explicit boolean incidence, support-aware rare incidence, sign-separated monetary checks with per-leg conditional quantile envelopes, and declared categorical TVD. Metric selection by physical dtype is dead (the audit's registry defect); an object-backed boolean gets its declared boolean contract. Tolerances are the chartered ones, NOT widened: support-awareness is a validity domain (insufficient support is receipted, never silently passed), and a tested one-sided hole — the run-7 160,667x class, where both origins carry ample support — fails under every profile. End-to-end (item 6): the pilot pipeline at fixture scale — stack, banked gap-fill with native ACS predictors, seeded clone attachment, one doctrine-mode PUF pass, completeness with proofs, terminal battery — proves the taxable-interest sentinel is healthy on ACS-origin rows, that skipping gap-fill is a named terminal predictor failure rather than a silent zero-fill, and that dropping a declared family from the surface fails the gate by name. The ratified 10% pilot configuration is declared as STACKED_PILOT_ACS_SAMPLE_FRACTION/SEED constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fail closed on null stacked recipient predictors * Fail closed on stacked completeness authority * Bind origin battery to declared authority * Make full clone attachment frame-identical * Enforce canonical donor byte identity * Accept canonical semantic object scalars * Deep-copy gate manifest details * Add round-two authority laundering regressions * Enforce canonical stacked-spine authority * Harden production authority signature guard --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run 6's production diagnosis: the transfer phase runs >7.5h on 16 cores, but observed container windows span 33 minutes to 8 hours — and while the 65 primary-QRF targets bank per-target, the 114 ACS-transfer targets restart from scratch on every preemption before the
transferredboundary lands. Five containers have died in that stretch.This extends the QRF directory's banking doctrine to the ACS transfer:
transferredcheckpoint identity is embedded per banked target — all six input pins, operator registries and order, engine version, take-up contract, materializer ledgers. Identity mismatch → ignored and rebuilt with a named receipt; torn/corrupt → rebuilt with a named receipt.transferredboundary checkpoint is unchanged (phase-complete only); the agreement gate remains uncached.Parallelism report (no config change): ACS targets are serial across targets; individual fits already run
n_jobs=-1with CPU-count prediction threads, so more cores help intra-target only and a material wall-time cut from cpu=32 is unproven.Verification: affected suite 129; #583 guard 495; input-consumer guard 169; full workspace 5,032 passed / 57 skipped / 0 failed; ruff + format +
git diff --checkclean.Part of #578 increment 2 hardening. Run 7 carries this if attempt 5b's window breaks; future pools complete under any window pattern regardless.
🤖 Generated with Claude Code