Skip to content

Add canonical US runtime stage contracts and docs#43

Draft
anth-volk wants to merge 10 commits into
mainfrom
us-runtime-stage-contracts
Draft

Add canonical US runtime stage contracts and docs#43
anth-volk wants to merge 10 commits into
mainfrom
us-runtime-stage-contracts

Conversation

@anth-volk
Copy link
Copy Markdown

Summary

Implements the canonical 9-stage US Microplex runtime taxonomy proposed in #20.

Closes #42.

This adds:

  • microplex_us.pipelines.stage_contracts for stable stage definitions, artifact contracts, validation placeholders, diagnostics, and resume posture.
  • microplex_us.pipelines.stage_manifest for saved-run stage manifests, validation evidence manifests, and Stage 6 entity checkpoint helpers.
  • artifact saving updates that write stage_manifest.json, source-plan, calibration-summary, PE entity checkpoint metadata, and validation evidence sidecars.
  • data-flow snapshot and PE-US-data checkpoint evidence updates that use the canonical Stage 9 validation/benchmarking stage.
  • Jupyter Book/Sphinx docs for the stage taxonomy, stage contracts/manifests, and API reference.

Out of scope

  • Automatic conditional execution or a scheduler.
  • A shared stage-validation runtime.
  • Full source-frame snapshotting for every raw input source.
  • Changes to modeling, synthesis, or calibration behavior beyond saved metadata/artifacts.

Validation

  • ruff check src/microplex_us/pipelines/stage_contracts.py src/microplex_us/pipelines/stage_manifest.py src/microplex_us/pipelines/artifacts.py src/microplex_us/pipelines/data_flow_snapshot.py src/microplex_us/pipelines/pe_us_data_rebuild_checkpoint.py src/microplex_us/pipelines/__init__.py tests/pipelines/test_stage_contracts.py tests/pipelines/test_stage_manifest.py tests/pipelines/test_data_flow_snapshot.py tests/pipelines/test_artifacts.py tests/pipelines/test_pe_us_data_rebuild_checkpoint.py tests/pipelines/test_backfill_pe_native_audit.py
  • PYTHONPATH=/private/tmp/microplex-comparison/microplex/src:/private/tmp/microplex-comparison/microplex-us/src python -m py_compile src/microplex_us/pipelines/stage_contracts.py src/microplex_us/pipelines/stage_manifest.py src/microplex_us/pipelines/artifacts.py src/microplex_us/pipelines/data_flow_snapshot.py src/microplex_us/pipelines/pe_us_data_rebuild_checkpoint.py src/microplex_us/pipelines/__init__.py
  • PYTHONPATH=/private/tmp/microplex-comparison/microplex/src:/private/tmp/microplex-comparison/microplex-us/src python -m pytest tests/pipelines/test_stage_contracts.py tests/pipelines/test_stage_manifest.py tests/pipelines/test_data_flow_snapshot.py (9 passed, 1 skipped; skipped parquet roundtrip because local env lacks pyarrow)
  • PYTHONPATH=/private/tmp/microplex-comparison/microplex/src:/private/tmp/microplex-comparison/microplex-us/src python -m pytest tests/pipelines/test_pe_us_data_rebuild_checkpoint.py::test_attach_policyengine_us_data_rebuild_checkpoint_evidence_updates_manifest tests/pipelines/test_pe_us_data_rebuild_checkpoint.py::test_attach_policyengine_us_data_rebuild_checkpoint_evidence_registers_calibration_only_runs tests/pipelines/test_backfill_pe_native_audit.py (6 passed)
  • PYTHONPATH=/private/tmp/microplex-comparison/microplex/src:/private/tmp/microplex-comparison/microplex-us/src jupyter-book build docs (passes; existing docs emit 206 warnings from pre-existing untoc'd pages/local absolute links)

Local limitation

The existing artifact save path still cannot run end-to-end in this local environment because pandas parquet writes require pyarrow or fastparquet, neither of which is installed here.

@anth-volk anth-volk force-pushed the us-runtime-stage-contracts branch from 86bc7ca to f29c42b Compare May 28, 2026 17:35
@MaxGhenis
Copy link
Copy Markdown
Contributor

I opened #45 as a small stacked follow-up for the run-contract/stage-manifest bridge.

Resolution there: keep this PR's 9 canonical stage IDs as the saved-run taxonomy, and map older operational run-contract IDs into that taxonomy when the dashboard reads run_summary.json. The dashboard preserves legacy labels as provenance via legacy_stage_id / legacy_completed_stages, so existing run summaries keep their debug value without creating a parallel lifecycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add canonical US runtime stage contracts, manifests, and docs

2 participants