Skip to content

CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) - #568

Merged
MaxGhenis merged 8 commits into
mainfrom
cg-tail-transfer-567
Jul 28, 2026
Merged

CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567)#568
MaxGhenis merged 8 commits into
mainfrom
cg-tail-transfer-567

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Phase 2 of the #567 charter, implementing the design adjudicated from the phase-1 diagnostic (receipts on #567): the dense netCG gap is a provable infeasibility — the QRF transfer truncates the donor CG tail at the ASEC topcode (frame max $1,999,998 vs donor max $22.1B), leaving design-weight CG × the hard 5× cap = $1.206T < the $1.271T target.

Four commits, in review order:

  1. Quarantine PUF mortgage outliers by field — the $10M mortgage screen becomes field-local instead of whole-row; recovers up to $98.2B of donor CG that was collateral attrition; per-field provenance accounting in the donor summary.
  2. Transfer the PUF capital-gains tail by weight split — the primary remedy: a deterministic, mass-conserving own-tail stratum through the existing clone seam (weight-splitting, never population-adding), consuming synthetic_tail_support_eligible (parsed since the disaggregation landed; first production consumer). Joint donor vectors — signed ST/LT together with linked Schedule-D quantities and lineage — never independently scaled legs (base-m collapses partnership_self_employment_net_earnings to signed near-cancellation (+47.7B/−48.1B legs) #432). Stage manifest with donor ids, boundary, masses, sha.
  3. Receipt PUF E01000 capital-gains divergence — audit artifact reconciling source E01000 vs the P22250+P23250 carrier concept across aggregate/synthetic/screen/donor/frame stages (5.1% raw divergence documented; no carrier change).
  4. Persist failed-release household weights — failed releases now write the final weight vector into the evidence dir (closes the completeness gap the diagnostic hit).

Verification: full packages/populace-build/tests/ shard green (verified independently by the dispatching session after rebase onto current main — the lane's stale-main mirror commit dropped cleanly as byte-identical to #565). +4,460/−88 across 26 files including two new test files (588-line tail-transfer suite, 339-line reconciliation suite).

Anti-charter honored: no loss shaping, no multipliers, no gate/envelope changes, no ratio changes, no re-disaggregation.

Exit criterion (build-level, next after merge): base-P3 rebuild → dense attempt where the 5× row ceiling exceeds the target with headroom and netCG passes the existing 0.25 gate on truthful inputs.

Fixes the netCG half of the dense-P3 publication path (#566 take-up design is the companion).

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits July 28, 2026 14:32
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confirm round 1 (HOLD, two blockers on the evidence commit):

- The persisted weight vector had no record identity: positional floats
  cannot be reattached to households once the run is gone, because the
  failed run minted no H5 and diagnostics record only n_records. The
  writer now persists the ORDERED household ids beside the weights
  (second npy), hashes the ordering, and binds both to the run's
  target-frame identity in the metadata.
- The write was unconditional at export_frame construction, so green
  runs wrote evidence files that no manifest accounts for. The call now
  lives inside the terminal_gate_failures branch only — the no-H5
  failure class this evidence exists for; late gates (reform smoke,
  take-up contract) raise after the H5 write, so their failures retain
  weights in the written dataset itself. Enforced structurally with the
  #443 AST-guard pattern: the sole main() call site must sit inside the
  terminal_gate_failures branch.
- Review test-quality finding folded: the SSI clone-2 test now mints a
  real weight-split clone — a new household carrying part of the source
  household's weight with total mass asserted conserved, duplicating
  the PUF-channel row the tail stage actually clones — instead of
  relabeling a full-weight row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round 2 pushed for both blockers: (1) the evidence writer now persists the ordered household ids beside the weight vector (second npy) with an ordering hash, and binds the pair to the run's target-frame identity — record-level reattachment is possible from the evidence alone; (2) the write moved inside the terminal_gate_failures branch — the no-H5 failure class it exists for — with green-run absence enforced structurally via the #443 AST-guard pattern (sole main() call site must sit in that branch; late gates raise after the H5 write, so their failures retain weights in the dataset). The test-quality finding on the SSI clone-2 test is folded too: it now mints a real weight-split clone (new household, weight subtracted from the source, total mass asserted conserved, PUF-channel row) rather than relabeling a full-weight row. Your High (#462 concentration not bounded by construction on the two flagged columns) stands as the stated build-time acceptance: the rebuilt frame's tail gate is the arbiter, and the register-or-redesign fork is pre-declared on #567. Full shard rerunning; targeted files green.

Confirm round 2 (HOLD, one blocker + two mediums):

- A green retry reusing --out/--release-id inherited the prior failed
  attempt's evidence files and certified an H5 whose manifest knows
  nothing about them. The green continuation now unlinks all three
  evidence files immediately after the batched gates pass, before the
  certified dataset write — evidence can only ever describe the run
  that is failing, never coexist with a certified release.
- The e2e identity assertion pinned only a nonempty base hash and the
  period; a truncated identity would have passed. It now rebuilds the
  expected identity from the captured materializer context (the
  established cache_context pattern) and asserts full equality.
- AST guard v2: the call must sit in the terminal_gate_failures IF BODY
  (an else-branch call would run on green), strictly before the
  branch's raise, and the green continuation must unlink stale evidence
  before write_dataset — ordering enforced structurally, matching how
  production is laid out.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round 3 pushed: (1) blocker — the green continuation now unlinks all three evidence files immediately after the batched gates pass and before the certified dataset write, so a green retry reusing --out/--release-id can never carry a superseded attempt's weights next to a certified H5; (2) the e2e identity assertion rebuilds the expected identity from the captured materializer context (the existing cache_context pattern) and asserts full equality — truncated identities now fail; (3) AST guard v2 enforces if-body placement (else-branch would run on green), evidence-before-raise ordering, and stale-cleanup-before-write_dataset on the green path. Builder test file green locally; full shard rerunning.

…lenames

Confirm round 3 (HOLD, certified-first blocker + guard advisory):

- The inverted sequence — certified green run, then a failed retry into
  the same --out/--release-id — wrote failed-attempt evidence beside the
  prior run's manifest and H5 (mkdir exist_ok retained them; the failure
  path never reaches the green cleanup). The run now refuses at setup
  when release_dir already carries release_manifest.json: certified ids
  are immutable, reruns mint a new id (every launcher stamps a fresh UTC
  timestamp) or the operator removes the directory deliberately. The
  failed-then-green cleanup path stands as adjudicated.
- The guard's cleanup check accepted any unlink anywhere after the
  branch; it now requires a For loop whose iterable names ALL THREE
  evidence filename constants with an unlink in its body, before
  write_dataset — decoy unlinks and dropped tuple entries fail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round 4: the certified-first sequence is closed by refusal per your prescription — _refuse_certified_release_dir_reuse raises at setup when the release dir already carries release_manifest.json (certified ids are immutable; launchers mint timestamped ids, deliberate reruns remove the dir), with a direct unit test. The failed→green cleanup stands as adjudicated. Guard v3 binds the cleanup to a For whose iterable names all three filename constants with an unlink in its body — decoy unlinks and dropped tuple entries now fail. Builder file green locally; full shard rerunning.

…coys

Confirm round 4 (HOLD, placement completeness + guard advisory):

- Refusal ran after checkpoint_root/artifact_root creation, so a
  hostile --checkpoint-root beneath releases/<id> mutated a certified
  directory before the raise, and the base download preceded the check
  entirely. Refusal now runs twice: once immediately when --release-id
  is given (before the base download — every launcher passes an id;
  the auto-generated id derives from the base digest, so that path's
  refusal is necessarily post-download) and once unconditionally the
  moment release_dir is computed, before any output mkdir. An AST
  ordering test pins both placements — the helper-only test cannot
  prove placement, per review.
- Guard v4: the cleanup iterable must BE a tuple naming all three
  evidence filename constants (subscript slices fail), and the unlink
  must be called on the loop target — in-loop decoys fail.

The Low TOCTOU advisory (concurrent same-id attempts racing exists())
is an accepted residual per review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Round 5: refusal now precedes every side effect — an early call the moment --release-id is known (before the base download; the auto-id path derives its id from the base digest so its refusal is necessarily later, but still before any output mkdir via the second, unconditional call at release_dir computation, ahead of _resolve_checkpoint_paths/all mkdirs). An AST ordering test pins both placements (2 refusal calls: one before _download_base_h5, one before every mkdir). Guard v4 requires the cleanup iterable to literally be a tuple naming all three filename constants (slices fail) with unlink called on the loop target (in-loop decoys fail). The Low TOCTOU race stands as the accepted residual you marked. Builder file green locally; full shard rerunning.

@MaxGhenis
MaxGhenis merged commit 66609b0 into main Jul 28, 2026
4 checks passed
MaxGhenis added a commit that referenced this pull request Jul 29, 2026
* Adult-care imputation refuses implausible donor knots

Base-P3 attempt 2 failed the adult-care signal gate with 12
pre_subsidy_care_expenses values above the $250,000 plausibility
ceiling. Checkpoint bisection traced the cause to two measured ASEC
childcare values ($730,000 and $360,000) that have sat latent in every
build's donor pool — their provenance is measured ASEC originals (their
mortgage values are trivial, ruling out the #568 quarantine), and
whether any recipient's quantile position reached those top knots was
grid luck. Build P3's tail clones shifted the recipient grid and 12
draws landed in the $360k-$730k interpolation region; base-P2 passed
the same gate over the same pool by geometry alone.

The ceiling's charter is refusing corrupted magnitudes, and a corrupt
magnitude is corrupt at the source: donors above the ceiling are now
refused as interpolation knots, with a logged receipt (count, values,
weight). The measured frame values stay untouched per the raw-only
doctrine — they are only barred from becoming other units' imputed
expenses. Draws are donor-value-bounded, so outputs can no longer
exceed the ceiling under any grid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Round 2: binding e2e, corrected narrative, receipt completeness, truthful docs

Confirm round 1 (HOLD — coverage and audit documentation):

- The poisoned-donor end-to-end binds the production call (mutation-
  checked: fails against the unscreened code): a $600k measured donor
  flows through with_us_adult_care_inputs; outputs stay bounded by the
  CLEAN donor maximum, the measured childcare column keeps its poisoned
  value untouched, incidence survives (usage computed pre-screen), and
  the receipt line is asserted. clean_mask2 asserted in the unit test.
- The narrative no longer claims all 12 draws landed between $360k and
  $730k — three were between the ceiling and $360k; the corrected
  wording is 'entered the >$250k top-tail interpolation region'.
- The receipt carries excluded_weight_share and retained_maximum, and
  states plainly that it is a deterministic build-log record, not a
  persisted artifact.
- The ceiling comment no longer claims $250k exceeds every measured
  donor value (it does not: $730k/$360k are measured), and the module
  docstring says the draw preserves the SCREENED level distribution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fold round-2 advisories: complete log receipt; exact narrative; incidence-bound e2e

The build-log line now carries excluded_weight_share and
retained_maximum; the unit-test docstring states the 3/9 split; the e2e
asserts exactly two expense carriers, so a screening-before-usage
mutation (usage 2/3 -> 1/3, one selected unit) fails the count.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Jul 29, 2026
…name

#568 added structural tests that locate the build body by AST, selecting the
function literally named main. This branch made main a thin entry point and
moved the body to _main, so those tests walked the four-line wrapper, found
none of the calls they check, and failed.

Neither change is wrong alone and the two merge cleanly, since they touch
different lines. Only the combination breaks, which is why it surfaced on CI
rather than in either branch.

All three by-name lookups now select _main: the certified-release-dir refusal
ordering, the final-household-weight evidence placement, and its bound cleanup
loop. Searched for every by-name reference rather than only the two CI named,
so the third could not fail on the next run.

Behavioral callers of main() are unaffected -- the wrapper delegates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant