Skip to content

Benchmark build: carry the disability inputs the references depend on (SSI criterion, SSDI months, permanent-and-total disability) #876

Description

@MaxGhenis

Summary

PolicyBench draws its households from the certified US build and scores frontier models against PolicyEngine references computed from exactly the inputs the build carries. On 2026-09-05 the 39-model board had to exclude eleven outputs from scoring because their references depend on inputs the June 2026 build (populace-us-2024-5da5a95-20260611) never carried, so the prompt could not list them (PolicyEngine/policybench#164, reference_exclusions.json). The v2 household draw needs a build that carries them.

What the June build lacks

Input State in the June build Consequence
meets_ssi_disability_criteria false for all 160,858 persons SSI payable only through age or blindness; six outputs excluded
months_receiving_social_security_disability absent in every build to date Medicare's under-65 route unreachable; five outputs excluded
is_permanently_and_totally_disabled never populated IRC 152 disabled-dependent age exemption and the IRC 22 credit unreachable
is_incapable_of_self_care absent until the 2026-07-23 builds (2,067 from PEDISDRS) IRC 21 CDCC disabled-dependent route

Builds from 2026-07-17 carry the SIPP-imputed SSI criterion (7,275 to 7,515 persons). #361 documents the sparse export dropping 94 no-formula inputs, including is_disabled, is_blind, employer_sponsored_insurance_premiums and the asset columns; the benchmark build must carry every no-formula input the engine reads.

Asks

  1. Certify a v2 candidate build that carries every no-formula PolicyEngine-US input (the Sparse-57k export drops 94 no-formula input columns from the dense parent (tips, overtime, disability flags, ESI premiums, rent, QBI basis) #361 fix), the SIPP SSI criterion, and is_incapable_of_self_care.
  2. Populate is_permanently_and_totally_disabled (impute from SSI/SSDI receipt or the SIPP donor) or document that the tax code's disability pathways are out of scope.
  3. Carry SSDI entitlement duration or start date (months_receiving_social_security_disability) with a defined reference date, if the SIPP donor can supply it; otherwise record the assumption explicitly so the benchmark can state it.
  4. Record provenance per disability fact (observed, imputed, unknown). An unknown must not export as an observed negative.
  5. Keep person_marital_unit_id in the export; the benchmark now builds marital units itself (head and spouse) after finding policyengine-core lumps a household into one unit when none is given.

Related: #424, #453 (SSI candidate undercount, child SSI absent), PolicyEngine/policyengine-us#9330 (disability input wiring audit), PolicyEngine/policybench#139 (v2 board).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions