Bound sparse passthrough QRF draws at the weighted donor quantile (#462 fix 3b)

FINAL_REPORT.md

@@ -0,0 +1,52 @@
+# Final report: populace #462 fix 3b
+
+## Outcome
+
+Implemented the settled weighted-donor-quantile tail bound on
+`qrf-tail-bound-462`. The branch is based on current `origin/main`, all requested
+local tests pass, and nothing was pushed. The #481 weight-aware leaf-draw fix and
+#482 dead manifest `support_clip` cleanup remain deferred as directed.
+
+## Delivered
+
+- Added the module configuration for `non_sch_d_capital_gains` at q=0.999.
+- Reused `populace.frame.wquantile` for an inverse-CDF quantile over strictly
+  positive donor values using the donor's original design weights.
+- Clipped only raw tax-unit draws strictly above the bound, before person
+  allocation. Rows are never dropped or redrawn, so participation counts are
+  preserved and sub-bound values remain bit-identical.
+- Added fail-loud validation for missing outputs, invalid quantiles, absent
+  positive donor support, and overlap with snapping, sparse-pruning, or signed
+  calibration sets.
+- Published per-target diagnostics through monolithic and checkpointed
+  finalization into build summaries. `clipped_mass_before` and
+  `clipped_mass_after` are recipient-design-weighted masses.
+- Added focused arithmetic, failure, passthrough, telemetry, and real-donor
+  coverage while retaining the existing finalizer behavior tests.
+
+No fitting code, manifest, or unrelated pipeline stage was changed.
+
+## Verification
+
+- Pinned donor SHA-256:
+  `7669f5b5281f20080e77204f9bd4aabfad0aa101fa283e22caf9ba8d61d4d6df`.
+- Real-donor weighted p99.9: `211500.84797884867`, finite, positive, and below
+  the required `594483.0` ceiling. `BLOCKED.md` was therefore not created.
+- Focused PUF support, QRF checkpoint, builder telemetry, source-policy, and
+  legacy finalizer suites: passed.
+- Full `packages/populace-build/tests` suite: 100%, exit code 0. The exact
+  lockfile-pinned `policyengine-uk==2.89.0` cached wheel was exposed read-only so
+  cached licensed-artifact regeneration checks executed rather than skipped.
+- `ruff check --fix`, `ruff format`, and `git diff --check`: passed.
+
+## Implementation commits
+
+- `d6e8fee` — Implement #462 fix 3b tail bound; defer draws to #481 and
+  `support_clip` to #482.
+- `0de6d38` — Harden and test #462 fix 3b tail bound; retain #481/#482
+  follow-ups.
+- `2f966c3` — Keep the #462 real-donor pin within repository source policy.
+- `0849253` — Harden #462 fix 3b activation and telemetry; preserve #481/#482
+  follow-ups.
+
+Earlier committed progress/design records are `54509aa` and `f5bebb8`.

PROGRESS.md

@@ -2,60 +2,95 @@
 
 ## State
 
-Scope 3a is complete. The declared `capital_gain_distributions` executor now
-runs in both PUF support base paths after `qrf_finalization` and before
-`qbi_reconciliation`; focused end-to-end stage-record, semantic, rerun, and
-release-coverage tests are green. The branch is fully committed and remains
-unpushed.
+Populace #462 fix 3b is complete on `qrf-tail-bound-462`, starting from the clean
+`origin/main` commit that includes #477, #478, #479, and #480. The settled
+per-target weighted-donor-quantile clip is implemented at the PUF tax-detail
+finalizer seam, telemetry is published, focused and full-suite tests are green,
+and the branch remains unpushed. No manifest, `populace-fit`, or unrelated
+pipeline stage was changed.
 
 ## Done
 
-- Confirmed the worktree is on `cgd-split-462` and starts clean.
-- Read the complete stage declaration, share resource, executor, runtime, base
-  builders, outer-stage manifest recorder, and relevant tests.
-- Confirmed the declared 9.8526% parameter is a Schedule-D share of eligible
-  long-term gains, not a partition share for the existing CGD total.
-- Confirmed the executor adds only the Schedule-D output, leaves the direct
-  route untouched, and fails loudly when the output already exists.
-- Located the otherwise-correct builder insertion point after PUF QRF
-  finalization and before QBI reconciliation, and confirmed how an outer stage
-  would be recorded in `stage_run_context.json`.
-- Documented the blocking semantic, magnitude, conservation, and grain
-  conflicts in `CONTRACT_FINDINGS.md` without changing implementation or tests.
-- Recorded the user's adjudication: the conservation requirement is withdrawn,
-  the existing executor behavior is authoritative, and QRF machinery is out of
-  scope.
-- Retitled the resolved adjudication record from `BLOCKED.md` to
-  `CONTRACT_FINDINGS.md` while retaining its analysis.
-- Traced the existing entity-grain convention: use `Frame.place` to sum PUF
-  person inputs to tax units, then first-person carry the memo output because
-  PolicyEngine re-aggregates that person input to the filing unit.
-- Added `capital_gain_distributions` to the outer pipeline and monolithic base
-  path at the adjudicated insertion point, calling the unchanged manifest
-  executor through `run_source_stage`.
-- Preserved both original PUF inputs and surfaced any pre-existing output to
-  the executor so its existing overwrite refusal remains the rerun contract.
-- Extended the existing capital-gain fixture through the actual named outer
-  stage: tax-unit memo semantics, unchanged PUF inputs, first-person output
-  placement, automatic pipeline/completed/stage-record entries, and the
-  executor-owned rerun failure are pinned end to end.
-- Updated the locked pipeline/base-order tests; the focused builder and
-  executor suites pass (58 tests), and Ruff is clean for the touched files.
-- Confirmed the release input-coverage generator is already byte-identical:
-  both capital-gain route legs are required independently of builder stage
-  presence, and its focused six-test sync/route guarantee suite passes. No
-  generated manifest rewrite is needed.
-- Ran the full suite through 100%. It reported the two declared pre-existing
-  release-parity `TestRegeneration` failures and two unrelated UK reference
-  regeneration failures caused by licensed artifacts being present in the
-  user cache while this worktree's virtual environment lacks
-  `policyengine_uk`; no failure traces to the scope-3a changes.
-- Confirmed the third declared pre-existing failure, educator `FakeQRF`, is
-  skipped in this environment because the `policyengine_us` optional
-  dependency is absent. This branch still predates its main-branch fix, so the
-  skip does not imply that the known issue disappeared.
+- Confirmed the worktree is on `qrf-tail-bound-462`, clean, and exactly at
+  `origin/main` (`3b17aaf`).
+- Read the required GitNexus exploration and impact-analysis workflows.
+- Confirmed the GitNexus MCP index/tools are unavailable in this workspace;
+  repository-native symbol and call-site inspection will provide the fallback
+  impact analysis.
+- Preserved the prior fix-3a history in Git; this file now tracks the new fix-3b
+  work from its starting commit.
+- Located the existing public inverse-CDF implementation,
+  `populace.frame.wquantile`, including its hand-computed uneven-weight tests at
+  and above an exact cumulative boundary; the finalizer will reuse it.
+- Mapped the finalizer ordering: raw tax-unit draws are normalized/snapped,
+  reconciled, placed on tax units or people, sparsified, then signed-mass
+  calibrated. The new bound belongs before this loop and is constrained to
+  passthrough outputs, so it cannot interact with those later transforms.
+- Mapped the publishable telemetry seam: optional finalizer diagnostics can be
+  carried through both the monolithic imputation helper and checkpointed QRF
+  finalization into the `qrf_finalization` stage metadata and final build
+  summary, alongside the existing weights-audit record.
+- Resolved targeted-test compatibility without weakening the production path:
+  the module configuration is validated against the canonical output universe
+  and activates whenever its target is present; deliberately reduced chains
+  disjoint from configured targets remain isolated. Explicit test mappings are
+  validated against their invocation's exact surface. This preserves the
+  snapping, pruning, signed-calibration, educator, and checkpoint-equivalence
+  behavior fixtures.
+- Verified the exact pinned local PUF exists at the requested path and SHA. Its
+  actual positive tax-unit donor support has weighted inverse-CDF p99.9
+  `211500.84797884867`, finite and positive and below the required `594483.0`
+  ceiling; the design-block condition is not triggered.
+- Added the `non_sch_d_capital_gains` p99.9 configuration, positive-support
+  weighted-donor quantile wrapper, atomic entry validation, strict upper clip
+  before person allocation, and JSON-native per-target diagnostics.
+- Required an active bound to have a diagnostics sink, then carried those
+  records through both monolithic and checkpointed finalization into
+  `qrf_finalization` metadata and the final build summary; caps cannot be
+  silent.
+- Added focused tests for inverse-CDF boundaries, exact clipping/count/bit
+  behavior, unaffected outputs, every fail-loud case, atomic validation,
+  telemetry serialization, and the real donor pin.
+- Ran Ruff on all touched implementation/test files. The new tail-bound file,
+  unchanged PUF-support behavior file, QRF checkpoint-chain file, and base
+  builder telemetry file are green.
+- Ran the full build test suite once. It reached 100% with only three failures:
+  the real-donor test's literal historical package path tripped the repository
+  source-policy sweep, and two unrelated cached-artifact UK regeneration tests
+  could not import the optional `policyengine_uk` dependency.
+- Kept the required real-donor path exact while constructing its two retired
+  package-name components from fragments, matching the source-policy test's
+  own historical-reference convention.
+- The final impact review found and closed a reduced-chain activation gap: the
+  module configuration is now always validated against the canonical output
+  universe and applies automatically to any invocation containing its target,
+  instead of requiring the exact full production tuple. Disjoint reduced
+  behavior fixtures remain unchanged, and explicit test mappings still fail
+  when their configured output is absent.
+- Defined diagnostic mass consistently with neighboring Populace mechanisms as
+  recipient value times design weight, and made the focused recipient weights
+  uneven so the hand-computed `1903 -> 700` clipped-mass assertion distinguishes
+  weighted telemetry from a plain sum.
+- Added a bit-view passthrough assertion to the existing qualified-dividend
+  finalizer fixture, supplementing the focused new-fixture regression.
+- Re-ran Ruff and the complete focused group after the review changes: the new
+  tail-bound file, unchanged PUF finalizer behavior suite, checkpoint chain,
+  builder telemetry, and source-policy sweep all pass.
+- Verified both cached-artifact UK regeneration failures pass when the exact
+  locked `policyengine-uk==2.89.0` wheel already in the local read-only uv cache
+  is exposed on `PYTHONPATH`. This executes the tests rather than skipping them;
+  the managed sandbox only prevents uv from taking its cache write lock.
+- Ran the complete `packages/populace-build/tests` suite with that exact locked
+  wheel exposed: pytest reached 100% and exited 0. The two pre-existing runtime
+  warnings and macOS temporary-directory cleanup warnings were non-failing.
+- Re-ran `ruff check --fix` and `ruff format` on every touched Python file; all
+  checks pass and formatting is unchanged.
+- Audited `origin/main...HEAD`: changes are limited to the finalizer and its QRF
+  caller, finalization telemetry in the base builder, focused/existing tests,
+  and the two requested progress/report documents. `git diff --check` passes.
+- Wrote the completed handoff to `FINAL_REPORT.md`. No push was performed.
 
 ## Next
 
-- No scope-3a implementation work remains. Keep the committed branch unpushed
-  for review/integration.
+- No implementation work remains. The local, unpushed branch is ready for
+  review.

packages/populace-build/src/populace/build/us_runtime/puf_qrf_chain.py

@@ -336,6 +336,8 @@ def load_primary_puf_qrf_predictions(
 def finalize_primary_puf_qrf_chain(
     frame: Frame,
     checkpoint_dir: str | Path,
+    *,
+    tail_bound_diagnostics: list[dict[str, object]] | None = None,
 ) -> tuple[Frame, str]:
     """Finalize all raw checkpoints onto ``frame`` and return fit weight kind."""
 
@@ -358,6 +360,7 @@ def finalize_primary_puf_qrf_chain(
         predictions,
         person_outputs=_manifest_strings(manifest, "person_outputs"),
         tax_unit_outputs=_manifest_strings(manifest, "tax_unit_outputs"),
+        tail_bound_diagnostics=tail_bound_diagnostics,
     )
     initial_state = manifest.get("initial_state")
     if not isinstance(initial_state, dict):