Skip to content

Canonical multispine operator ordering: assemble spines first, then clone/impute/derive/seed/simulate/calibrate #395

Description

@MaxGhenis

The canonical operator ordering on the multispine

Max's architecture directive (2026-07-11): build stages are operators on ONE assembled multispine frame, in this order — not per-source pipelines that each re-implement enrichment:

  1. Harmonize (per-source, the only per-source layer): map each survey's raw observations into shared harmonized columns with universe metadata (e.g. the six-question disability battery from both ASEC PEDIS and ACS DEAR/DEYE/DREM/DPHY/DDRS/DOUT).
  2. Assemble the multispine: all survey spines (ASEC, ACS 1-yr, future: ACS 5-yr, SIPP) into one frame with spine tags and mass allocation.
  3. PUF cloning as a frame operator: fused tax-detail twins for records on EVERY spine, conditioned on the harmonized demographic+income surface — the ACS spine gets direct fusion instead of QRF transfer from the ASEC donor (retires the two-hop imputation for PUF-origin detail).
  4. Cross-source imputations (SCF wealth, SIPP tips, MEPS, …) once, on the pooled frame.
  5. Deterministic derivations (is_disabled via the SSI alignment rule over the harmonized battery, structural own_children_in_household, …) once, spine-agnostically.
  6. Runtime seeds (take-up draws) — already frame-level today.
  7. Engine materialization / microsimulation for target measures.
  8. Calibration, then selection/export, with gates at each stage boundary.

Where today deviates

The ASEC×PUF donor pool is built first (steps 3–4 run on the ASEC spine only), and the ACS spine attaches afterward via QRF transfer from that donor (acs_transfer.py). This is the shipped v1 (launch) architecture; it is honest and provenance-tracked, but PUF-origin detail on ACS carries two modeling hops, and per-source derivation risks rule drift (being fixed frame-level for disability/children on the buildl-acs-spine branch).

Scope of this issue

  • Migrate PUF cloning from the ASEC base-pool build to a multispine frame operator (design: conditioning surface, clone mass semantics per spine, memory plan — ACS 1-yr doubles to ~2.6M rows).
  • Migrate SCF/SIPP/MEPS imputation stages to run post-assembly on the pooled frame.
  • Certified-path guard throughout: each migrated stage must prove bit-identical output on the ASEC spine vs the current pipeline before the per-source implementation is retired (same guard pattern as the disability-derivation refactor).
  • Sequencing: post-launch; the v1 transfer architecture ships Monday with its provenance register and deferred-inputs list documenting exactly what the reorder improves.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions