Skip to content

Drop raw-source and build-intermediate columns from the published populace-us artifact #27

Description

@MaxGhenis

Summary

The published populace-us artifact should store PolicyEngine-recognized inputs plus explicit structural/geo columns, not raw source-survey fields or build scratch columns.

Drop (raw source + build intermediates)

  • Raw CPS ASEC columns: A_AGE, A_FTPT, WSAL_VAL, SEMP_VAL, WC_VAL, DIS_VAL1, etc.
  • Raw SCF columns such as scf_net_worth, scf_bank_account_assets, and other scf_* source fields once the PE inputs they feed are set.
  • Build scratch: _half, _orig_household_id, *_is_puf_clone, new_tax_unit_id, and redundant intermediate weights if they duplicate calibrated weights.

Keep

  • Geo identifiers used by the local-area pipeline: block_geoid, tract_geoid, congressional_district_geoid, cbsa, etc.
  • Structural keys and calibrated weights.

Acceptance criteria

  • The export surface has an explicit structural/geo allow-list.
  • Raw source and scratch columns are removed upstream before final artifact writing.
  • A gate fails when an artifact contains columns outside PE variables plus the allow-list.
  • Tests cover a raw-source column, a scratch column, and a retained geo/structural column.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions