Restore eCPS export and entity ID parity#112
Merged
Merged
Conversation
MaxGhenis
added a commit
that referenced
this pull request
May 30, 2026
Add microunit as a dependency and route the reconstruction-from-scratch tax-unit path through microunit.construct_tax_units when the person frame carries microunit's raw CPS input columns (PH_SEQ, A_LINENO, A_MARITL, A_SPOUSE, PEPAR1, PEPAR2, A_EXPRRP). When those columns are absent -- the current production case, since microplex's reconstruction frame collapses relationship_to_head and drops the spouse/parent pointers -- the new USPipeline._build_policyengine_tax_units_via_microunit returns None and the legacy role-flag reconstruction runs unchanged. The authoritative-ID path (#112) is never routed here. Net effect is behavior-preserving on today's data: the delegation stays inert until an upstream change threads CPS columns through to entity construction. microunit IS eCPS's tax-unit engine, so activating the delegation converges microplex's tax units toward eCPS's; any resulting loss movement is an entity-convergence effect and must be interpreted as such, not as a quality win (see #113). Adds tests/pipelines/test_us_microunit_delegation.py (4 passing); ruff clean. Implementation produced by the parallel wire-microunit agent; verified (ruff + delegation tests) and committed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
May 31, 2026
…ing; part of #113) (#114) * Delegate PE tax-unit reconstruction to microunit (part of #113) Add microunit as a dependency and route the reconstruction-from-scratch tax-unit path through microunit.construct_tax_units when the person frame carries microunit's raw CPS input columns (PH_SEQ, A_LINENO, A_MARITL, A_SPOUSE, PEPAR1, PEPAR2, A_EXPRRP). When those columns are absent -- the current production case, since microplex's reconstruction frame collapses relationship_to_head and drops the spouse/parent pointers -- the new USPipeline._build_policyengine_tax_units_via_microunit returns None and the legacy role-flag reconstruction runs unchanged. The authoritative-ID path (#112) is never routed here. Net effect is behavior-preserving on today's data: the delegation stays inert until an upstream change threads CPS columns through to entity construction. microunit IS eCPS's tax-unit engine, so activating the delegation converges microplex's tax units toward eCPS's; any resulting loss movement is an entity-convergence effect and must be interpreted as such, not as a quality win (see #113). Adds tests/pipelines/test_us_microunit_delegation.py (4 passing); ruff clean. Implementation produced by the parallel wire-microunit agent; verified (ruff + delegation tests) and committed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Depend on microunit from PyPI (>=0.1.0) instead of the git pin microunit 0.1.0 is now published to PyPI, so drop the pre-PyPI git+https commit pin in favor of a standard version constraint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Harden microunit delegation: defensive filing-status normalize + order test - Route microunit's filing_status through _normalize_policyengine_filing_status so the delegated path cannot diverge from the legacy paths if microunit ever changes its spelling/casing (today the vocabularies already match). - Add a regression test feeding rows out of PH_SEQ/A_LINENO order, asserting correct unit/role/filing assignment — locks in microunit's input-row-order contract that the positional TAX_ID mapping relies on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced May 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
is_household_headfromrelationship_to_headwhen PE entity tables do not already carry the explicit input.family_id,spm_unit_id, andmarital_unit_idcolumns during PE entity-table construction, normalizing IDs only when raw IDs repeat across households.Why
The no-ACS eCPS-shaped comparison showed Microplex still missed one non-formula eCPS-exported column,
is_household_head. The remaining eCPS-only columns are PE formula-owned and should stay excluded under the no-formula-export contract. The same comparison also exposed entity structure drift, especially SPM/family/marital group construction; preserving complete source IDs is the safest first step toward source-stage parity.Validation
uv run --python 3.13 --extra dev --extra policyengine pytest tests/policyengine/test_us.py::TestPolicyEngineUSProjection tests/pipelines/test_us.py::TestUSMicroplexPipeline -k 'policyengine_entity_tables or export_variable_maps or time_period_arrays or group_tables or tax_unit'uv run --python 3.13 --extra dev --extra policyengine ruff check src/microplex_us/policyengine/us.py src/microplex_us/pipelines/us.py tests/policyengine/test_us.py tests/pipelines/test_us.pyuv run --python 3.13 --extra dev --extra policyengine ruff format --check src/microplex_us/policyengine/us.py src/microplex_us/pipelines/us.py tests/policyengine/test_us.py tests/pipelines/test_us.pyis_household_headis now present; the only remaining eCPS-only variables are the nine PE formula-owned columns intentionally excluded from Microplex export.