Skip to content

UK dataset sizes: informed L0 and exact-count candidates (#355) - #877

Draft
juaristi22 wants to merge 1 commit into
uk-rowwise-candidate-762-bfrom
uk-dataset-sizes-355
Draft

UK dataset sizes: informed L0 and exact-count candidates (#355)#877
juaristi22 wants to merge 1 commit into
uk-rowwise-candidate-762-bfrom
uk-dataset-sizes-355

Conversation

@juaristi22

Copy link
Copy Markdown
Collaborator

The UK rowwise driver currently exports every cloned household. This adds --dataset-households to request a compact candidate after the dense solve, using contribution-informed L0, exact-count sampling, and a frozen-support refit.

Stacked on #870 (uk-rowwise-candidate-762-b) so the diff contains only the size-selection increment. Refs #355 and #665; this does not close #355's certification and default-switch requirements.

Behavior

  • Keep pool generation separate from output size: the existing K=15, grain_equal, 1,500-epoch defaults and local gates remain.
  • Initialize L0 using target-contribution shares and protect each nonzero target's largest absolute weighted carrier. The prior initializes search; it is not a top-k selector.
  • Reuse exact-count Sampford sampling and the shared normalized Horvitz–Thompson refit baseline. The stretch multiplier remains 10 relative to that inclusion-adjusted baseline; the manifest explicitly records this reference.
  • Freeze full-pool target contributions, preserve linked households, benefit units and people, and rerun selection inside each holdout fold.
  • Record requested/realized sizes, selection provenance, dense/refit diagnostics and Logbook receipts.

Review boundary

These are candidate artifacts. Size requests refuse --release-candidate and remain unreleasable until matched comparisons and the promotion scorecard are adjudicated. No dataset registry, production pointer or downstream default changes. Full builds still require the dense build's peak memory and add selection/refit work; savings apply to the exported dataset and downstream use.

The prior mapping and inclusion-adjusted stretch reference need measured UK evaluation before release. No licensed full-data build or reform scorecard was run. The implementation plan and operating instructions are in docs/uk-dataset-size-plan-355.md.

Validation

  • Calibration, UK rowwise runtime/driver, doctrine, dense assembler and country-spec regression run: 433 passed.
  • Additional identity/seed, US exact-count compatibility and UK runtime run: 66 passed (overlapping the regression run).
  • Final size-specific and dry-run checks: 16 passed.
  • Repository-wide Ruff check, changed-file formatting, diff whitespace and CI test inventory verification passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant