Skip to content

QRF donor contains aggregate-scale PUF rows (single-record home_mortgage_interest up to $236B); support_clip donor_realized is vacuous against them #516

Description

@MaxGhenis

Found during #511 forensics; latent hazard, not the realized driver of that row (base-O pool is clean of it — receipts below).

What's in the donor

Measured via puf_tax_unit_donor_from_arrays on the pinned artifact the certified O base consumed (base-o summary puf_h5policyengine_us_data/storage/puf_2024.h5, sha 7669f5b5…, 211,677 donor rows):

  • 3,066 rows carry home_mortgage_interest ≥ $10M — up to $235.97B on a single record (three identical $236.0B rows at weight 0.13 each; $177.0B ×2; $22.0B × w1.08; $3.7B × w11). Weighted, these rows hold $2,947B of phantom mass against $418B for the other ~94k carriers. TY2015 E19200 for ALL returns was $304.5B — single-record values of this scale are not tax data; they look like IRS disclosure-aggregate rows (or their clones) whose amounts were never brought back to per-return scale.
  • The same rows poison the structural columns: donor-wide implied first-mortgage rate = 31.9% of balance (34% for 2005-2010 originations, 50% for 2010-2016) vs ~4% reality.

Why nothing burned on O-1

Why it still deserves a fix

  1. support_clip range: donor_realized is vacuous for every column an aggregate-scale row touches — the realized range extends to $236B, so the clip can never catch a runaway draw. That defeats the backstop exactly where it's most needed.
  2. It's a seed lottery with a tiny-but-nonzero jackpot (QRF amount forests lose donor design weights after the n-of-n bootstrap: extreme-tail draws are a seed lottery #481 mechanism): one included copy of a $236B row in a leaf is catastrophic for any sparse column these rows carry.
  3. Every puf_tax_detail output column shares the donor, so the exposure is not mortgage-specific.

Ask

Either screen aggregate-lineage rows out of the QRF donor path (they're identifiable by scale/weight; the disaggregate_aggregate_records op already knows the RECIDs upstream), or make them sane at ETL (per-return-scale values after disaggregation), and make support_clip robust to them (e.g., clip range from the non-aggregate donor support). Relates #481 (weight-loss lottery), #487 (donor uprating class), #511 (discovery context).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions