The stacked spine: one origin-labeled spine, banked gap-fill, null-vs-zero doctrine, by-origin battery (#578 revision pilot) - #616
Merged
Conversation
…ine (#578 revision item 1) The ratified increment-2 revision removes the two-spine agreement seam: assemble_stacked_spine() draws a whole-household ACS sample (exact-count rule floor(fraction * eligible), seeded, deterministic over equal frames), reuses the reviewed assemble_spines seam unchanged, and freezes a stacked- spine manifest binding fraction, seed, the realized selection digest, and per-arm weight-harmonization receipts to the live rows. Origin labels survive as the ordinary support-channel columns. Weight harmonization follows the two-arm P-lineage precedent (each arm receives a declared share of the anchor mass; the seeded sample's inverse-fraction upweighting falls out of the mass allocation and is receipted, not assumed). validate_stacked_spine_frame() fails closed on any mutation of the sample, counts, digest, shares, or manifest. stacked_spine.py is registered as a source-spine provenance OWNER in the blindness guard (a reviewed contract change): stacking, gap-fill routing, activation authority, and the by-origin battery are origin-aware by charter, so population operators can stay spine-blind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…UF pass (#578 revision items 2-3) The doctrine (audit item 1, binding): absence stays null until an authorized stage fills it. finalize_us_puf_tax_detail_predictions gains an absent_cells policy — the default reproduces the historical global fillna(0.0) byte for byte for the retiring lineage, while PUF_ABSENT_CELLS_PRESERVE_NULLS (hard-wired in every stacked entrypoint) creates missing columns as null, keeps coercion from manufacturing values, and scopes the donor-rate sparsification rewrites to the PUF clone arm so no boundary converts absence into an observed zero. The audited defect is pinned as a regression test in both directions. Strict recipient predictors (audit items 2-3): prepare_us_puf_tax_detail_chain_inputs / impute_us_puf_tax_detail_support gain require_complete_recipient_predictors — recipient features are built null-preserving and a missing predictor value on any recipient row is a named terminal failure instead of the silent zero-fill that collapsed ACS clone-1 draws to ~$2 degenerates in run 7. Gap-fill (item 2): gap_fill_stacked_spine() fills survey-specific fields cross-origin on the unclonled stacked spine under DECLARED activation authority — per direction, every target column must exist, donor-origin rows must observe it completely, and every null must lie on the declared recipient origin, verified before and after the fit (donor cells must be byte-identical). Donors are this owner's projection of the donor origin's native rows (authoritative rows, audit item 3); the spine-blind transfer_acs_inputs machinery and the #608 per-target banking store are reused unchanged, and the native ACS aggregates (acs_interest_dividend_rental_income et al.) serve as recipient predictors through the existing feature surface. Single PUF pass (item 3): clone_us_frame_for_puf_support gains a seeded whole-household clone attachment (floor(fraction * eligible), explicit seed, manifest-bound selection digest, pair-symmetric half weights, fraction=1.0 byte-identical to the full clone) and run_stacked_puf_pass() runs the primary QRF once over the stacked spine after gap-fill under both doctrines, with per-origin recipient receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d-to-end (#578 revision items 4-6) Completeness gate (item 4, audit item 5): stacked_completeness_gate() proves every declared entity/family/target is filled or carries an explicit AbsenceProof naming the origin channel and clone role whose cells may be null (with the reason receipted). A missing column and every unproven null are named terminal failures with per-origin, per-role counts — the check that would have caught run 7's silent 58-target skip, now paired with the preserve-nulls doctrine so a skipped family presents as nulls rather than observed zeros. By-origin battery (item 5): by_origin_battery() replaces spine-vs-spine agreement with the same statistics scoped to gap-filled families and per-family DECLARED metrics — explicit boolean incidence, support-aware rare incidence, sign-separated monetary checks with per-leg conditional quantile envelopes, and declared categorical TVD. Metric selection by physical dtype is dead (the audit's registry defect); an object-backed boolean gets its declared boolean contract. Tolerances are the chartered ones, NOT widened: support-awareness is a validity domain (insufficient support is receipted, never silently passed), and a tested one-sided hole — the run-7 160,667x class, where both origins carry ample support — fails under every profile. End-to-end (item 6): the pilot pipeline at fixture scale — stack, banked gap-fill with native ACS predictors, seeded clone attachment, one doctrine-mode PUF pass, completeness with proofs, terminal battery — proves the taxable-interest sentinel is healthy on ACS-origin rows, that skipping gap-fill is a named terminal predictor failure rather than a silent zero-fill, and that dropping a declared family from the surface fails the gate by name. The ratified 10% pilot configuration is declared as STACKED_PILOT_ACS_SAMPLE_FRACTION/SEED constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 5, 2026
juaristi22
added a commit
that referenced
this pull request
Aug 6, 2026
…ning), #616 Brings Max's #617 adversarial-review fixes (preflight alias rejection, fail-closed SDC floor, per-artifact audit isolation, strict JSON) and the merged #610 base under this branch; the PR retargets main. # Conflicts: # changelog.d/609-uk-weighted-integrity-gates.added.md # changelog.d/612-uk-frame-foundations.added.md # packages/populace-build/src/populace/build/uk_runtime/hmrc_source_contract.py # packages/populace-build/src/populace/build/uk_runtime/terminal_gates.py # packages/populace-build/src/populace/build/uk_runtime/weighted_integrity.py # packages/populace-build/tests/test_uk_efrs_weighted_totals.py # packages/populace-build/tests/test_uk_frame_linkage_preflight.py # packages/populace-build/tests/test_uk_rowwise_dataset.py # packages/populace-build/tests/test_uk_terminal_gates.py # packages/populace-build/tests/test_uk_weighted_integrity.py # packages/populace-data/src/populace/data/contract.py # tools/build_uk_national_dataset.py # tools/preflight_uk_frame_linkage.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run 7's agreement failure traced to the two-spine seam itself: the primary PUF-QRF ran on 2.1M ACS clone-1 rows with zero-filled target-like predictors (degenerate ~$2 draws), finalization converted the remaining absence into observed zeros, and the null-activated ACS transfer silently skipped all 58 PUF targets. Per the ratified #578 revision charter, this pilot removes the seam rather than patching it.
Item 1 — the stack:
assemble_stacked_spine()draws a seeded whole-household ACS sample (pilot fraction 0.10, seed 578; exact-count floor rule) and stacks it with ASEC into one origin-labeled spine under the two-arm P-lineage weight math (w′ = w × share × M_anchor / M_arm, realized scale factors receipted). A frozen manifest binds fraction/seed/counts/selection digest; five mutation classes fail closed.Items 2–3 — gap-fill + doctrine + one PUF pass: cross-origin gap-fill runs pre-clone under declared activation authority (donor rows must observe the target completely; every null must lie on the declared recipient origin — verified before and re-verified after), through the unmodified spine-blind transfer with #608 per-target banking, using the audit's named native predictors. The null-vs-zero doctrine lands in
puf_support.py:absent_cells=preserve_nullsscopes the finalization fillna andrequire_complete_recipient_predictors=Truemakes a null predictor a named terminal failure — run 7's zero-fill is pinned byte-for-byte as a regression in both directions. The PUF pass then runs once over the stacked spine with seeded clone attachment (fraction 1.0 byte-identical to the full clone, asserted).Item 4 — completeness gate: every declared target filled or carrying an
AbsenceProof(origin channel × clone role × reason); run 7's silent 58-target skip presents as unproven nulls and terminates by name.Item 5 — by-origin battery: agreement statistics reborn per-origin with declared per-column metrics (boolean_incidence / rare_incidence / monetary_sign_separated / categorical_tvd) — dtype-dispatch dead (the CHAMPVA defect pinned), tolerances NOT widened, insufficient support receipted rather than passed, and the 160,667× one-sided-hole class failing under every profile.
Item 6 — tests: 40 in test_us_stacked_spine.py including the fixture e2e (stack → banked gap-fill → attachment → doctrine PUF pass → completeness → terminal battery) asserting the taxable-interest sentinel is healthy on ACS-origin rows and that skipped gap-fill is a named failure, never a silent zero.
Memory envelope (planning receipts): pilot ≈49 GB proportional peak (7.2× headroom), full fraction ≈241 GB (1.5×), with two structural savers below that. Deferred to production wiring: pool-tool adoption, the full 131-column battery registry, clone-2 tail integration, the SSI rerun.
Verification: #583 guard 495 exactly; full workspace 5,110 passed / 57 skipped / 0 failed; ruff + format +
git diff --checkclean; legacy two-spine path byte-identical under defaults.Part of the #578 increment-2 revision (charter: issue comment 5185931433).
🤖 Generated with Claude Code