populace-frame kernel: Frame, typed weights, strata, links, accounting, units, RulesEngine protocol

[MaxGhenis]

Implements DESIGN.md sequencing step 1: the kernel package every operator in the populace stack works on — now shipped as the first shard of the populace namespace.

Packaging: the populace namespace

packages/populace-frame is a PEP 420 namespace shard: src/populace/frame/ with no top-level populace/__init__.py, so populace-frame, populace-fit, and populace-calibrate install side by side and a populace metapackage can pin the constellation. The distribution is populace-frame 0.1.0 (hatch wheel ships src/populace); the import is populace.frame; extras are populace-frame[us] and populace-frame[policyengine]. Verified by building the wheel (no populace/__init__.py inside) and importing from a clean venv. microplex keeps its own repo and brand — it is the engine, not a shard.

What's in populace.frame

weights — WeightKind (design | importance | calibrated) and an immutable Weights vector validated on construction: finite, non-negative, never all zero. Kinds only move forward (design -> importance -> calibrated); assert_kind_transition is the single authority for that rule, and assert_mass_conserved reports both totals on violation.

schema — EntitySchema (person + group entities, with the person_{group}_id / {group}_id linkage convention), the new LinkSpec (below), and VariableMetadata (entity, dtype kind, period semantics).

bundle — Frame, the atom of the stack (renamed from WeightedBundle, no legacy alias). The name is the survey-statistics term: a sampling frame is the list of units a sample is drawn from and the thing weights refer back to — a Frame is that object made executable. Kernel invariants enforced on every construction with precise errors: exact group-id partitions (orphans and dangling membership both named), unique person ids, weight-vector lengths, person-aligned strata, and globally unique column names across entity tables (the flattening rule). Operations all return new, re-validated frames: with_weights (kind + optional mass-conservation gates), broadcast (group column → persons), concat (pool-strata assembly: strata must differ or id spaces be disjoint; integer ids shift past collisions; mass is preserved exactly; union kind is the further of the two), select (subset persons, prune groups, re-validate), plus stratum_mass and accessors.

links (documented placeholder) — LinkSpec(name, left_entity, right_entity) declares many-to-many associations (e.g. a jobs link between persons and firms) on EntitySchema.links, validated for name uniqueness, entity-name collisions, and declared sides. A Frame accepts link tables in tables keyed by link name, requires both linked entities' id columns, and validates every referenced id against the linked tables. Link tables are join tables, not entity tables: exempt from the flattening rule, and refused by select/concat until the full link operator lands. Contract-tested with a tiny person×firm jobs link.

accounting — wsum, wmean, wquantile (inverse-CDF), wmedian, gini (Lorenz trapezoid, with the negatives= handling), groupby_wsum, absorbing microdf's reason to exist. Columns weight through effective entity weights: explicit vectors win; person columns inherit the weighted group entity through membership; unweighted group entities collapse member-constant person weights (refusing if members disagree). NaN propagates — no silently partial aggregates.

units — the proven CPS ASEC unit-structure operator, returning a validated Frame (person + household, tax_unit, spm_unit, family, marital_unit). Tax units stay delegated to microunit (optional extra populace-frame[us]); SPM/family/marital systems are deterministic factorizations with the _validate_partition guard intact. Household weights are typed Weights, accepted per-household or per-person (constant-within-household, collapsed).

rules — the RulesEngine protocol exactly as in DESIGN.md (variable_entity, variable_dtype, entity_schema, materialize, export_contract, write_dataset) plus the ExportContract dataclass (required / forbidden / optional / formula_owned_excluded, from_path). adapters/policyengine_us.py implements the protocol with lazy engine imports (populace-frame[policyengine] extra): materialization through Microsimulation, exports as gated, round-trip-verified USSingleYearDataset files with the frame's household weights materialized as household_weight.

Behavioral contract suite (process rule 1)

tests/test_contracts.py states the platform guarantees as tests: corrupt weight vectors can never enter; kinds never move backward; mass-conservation errors carry both totals; frames reject orphan ids, duplicate global columns, misaligned strata; concat of two strata preserves mass exactly; select re-validates; wsum equals the hand-computed weighted sum; gini is 0 under equality and (n-1)/n under total concentration; unit assignment partitions exactly; declared links validate their tables against the linked tables' ids; the adapter satisfies the protocol; exports round-trip through the rules adapter. The populace-fit contract ("weighted fit shifts draws toward weighted truth") is encoded as a skipped placeholder until that package lands, so the suite carries the full contract from day 1.

Docs

DESIGN.md's kernel and Naming sections and the READMEs move to the populace naming: shard distributions populace-frame / populace-fit / populace-calibrate, imports populace.frame / populace.fit / populace.calibrate, a populace metapackage for the constellation, and microplex staying its own repo/brand for the engine. Other charter sections are untouched.

Verification

• Workspace (uv sync --all-packages): 139 passed, 3 skipped; ruff check clean (with populace declared first-party for isort — PEP 420 defeats src-layout inference).
• Wheel built and inspected: ships populace/frame/** with no populace/__init__.py; clean-venv install imports populace.frame and constructs a Frame with a validated jobs link.
• The package imports without policyengine_us installed; engine-backed methods raise ImportError naming the extra.

🤖 Generated with Claude Code

---

Adversarial review applied (2026-06-10)

An independent clean-room review reproduced three false-guarantee bugs and several gaps; all addressed on this branch:

• C1 typed weights are authoritative over a stale household_weight column at export.
• C2 with_weights mass policy is now an explicit required decision (mass="conserve" or MassChange(...)); silent mass loss is impossible.
• H1 wquantile/wmedian propagate NaN like the other aggregates.
• H2 RulesEngine resolves variable_metadata (entity+dtype+period) and enumerates variables(); reform/multi-period deferred explicitly.
• Cleanups: dropped unused pydantic dep; namespace-package contract test; export round-trip asserts dtypes.

Charter (DESIGN.md) revised for the longitudinal contradictions, shard rationale + versioning mechanism, a promoted evaluation section, and a measurable three-part disclosure criterion.

Deferred (lower severity, tracked): engine-installed CI job (H3), weight-kind laundering (H4), and assorted M/Low items left as # REVIEW notes.