fix(#323): Mali pid was keyed on the RESPONDENT, not the person — recovers 32,026 people#616
Merged
Conversation
…ored)
Mali declares household_roster and individual_education on (t, i, pid), but
2014-15 wired `pid` to `s01q`. Per EACIIND_p1.dta's own variable labels:
s01q00 = "Numero d'ordre" -- the person's roster line
s01q = "Code du répondant" -- the line of whoever ANSWERED for the HH
`s01q` is a HOUSEHOLD-level fact stamped onto every member row. It is constant
within 3,335 of the 3,804 households, 28,054 of 37,175 rows carry s01q == 1 (the
head answered), and every household's s01q value is itself one of that
household's own s01q00 lines -- it is a reference INTO the person numbering, not
a person id.
So the declared index was not unique in the source:
uniq (grappe, menage, s01q) = 5,149
uniq (grappe, menage, s01q00) = 37,175 (= one row per person, exactly)
_normalize_dataframe_index then reduced the declared index with
groupby().first(), keeping ONE PERSON PER HOUSEHOLD and deleting 32,026 people
with no signal. household_roster reported a mean household size of 1.354.
Fixing the identifier -- rather than declaring a reducer for it, which would only
sign the corpse (no reducer is correct: `first` keeps one person, `sum` is
meaningless on Sex) -- restores the table:
household_roster 2014-15: 5,149 -> 37,175 rows (1.354 -> 9.773 mem/HH)
individual_education 2014-15: 2,093 -> 4,097 rows (= the 4,097 source
rows with non-null
s02q23)
and makes (t, i, pid) unique IN THE SOURCE, 37,175/37,175. Note the returned
frame tested as index-unique both before and after -- that is the disease, not
the cure: the collapse is what made it unique. All measurements are cold builds
(LSMS_NO_CACHE=1); the L2 parquet is written post-collapse and cannot show the
loss.
This also re-converges household_roster.pid with _/people_last7days.py, which
already keyed on (grappe, menage, s01q00) and asserted uniqueness -- the correct
formula was in the tree the whole time.
Verified against the raw .dta: HH (grappe=1, menage=3) has 9 members, ages
[0,1,2,3,5,7,20,28,43], 6M/3F -- the API now returns exactly that, with pid
1003001..1003009; HH (1, 6) likewise returns all 15 members. Derived
household_characteristics picks the fix up for free (log HSize for those two
households = ln(9) and ln(15)).
Blast radius checked across every Mali (wave x table) declaring pid: 2017-18
(`codeid`) and 2018-19 / 2021-22 (`numind`) were already unique in source and are
unchanged. 2014-15 was the only affected wave.
Config-only; no lsms_library/*.py touched (core is owned by #614 et al).
Scope: the pid identifier only. The interview_date/`visit` half of the original
fix/323-mali branch needs a new {const: value} core primitive and ships
separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…st7days.py:48 Records that (grappe, menage, s01q00) was not invented for this fix: the wave's own people_last7days.py already built pid that way and asserted (t,i,pid) uniqueness. data_info.yml had simply drifted out of that keyspace. Marks the quantity as REUSE, and records the §4 invariant this task turns on -- a declared index must be unique in the SOURCE, because the L2 parquet is written post-collapse and a warm read cannot see the loss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two source columns, one character apart. 32,026 people.
Fixes the GH #323 defect in Mali
household_rosterandindividual_education. Config-only — zero changes tolsms_library/*.py.Part of the #323 consolidation (
slurm_logs/DESIGN_grain_collapse_sites_2026-07-13.org). This is the case the whole no-aggregation policy is argued from — and it turns out the fix is a typo, exactly as the policy predicts: duplicates on a declared index mean the identifier is broken, so fix the identifier. Never declare a reducer.The bug
Mali/2014-15/_/data_info.ymlwiredpid's third component tos01q. Straight fromEACIIND_p1.dta's own variable labels:s01q00→ "Numero d'ordre" — the person's own roster line. ✅s01q→ "Code du répondant" — the roster line of whoever answered for the household. ❌s01qis a household-level fact stamped onto every member row. It is constant within 3,335 of 3,804 households; 28,054 of 37,175 rows carrys01q == 1(the head answered); and every household'ss01qvalue is itself one of that household's owns01q00lines — it is a reference into the person numbering, not a person id.So the declared index
(t, i, pid)was really keyed by who answered.groupby().first()kept one person per household and the rest vanished silently.The fix
In two tables —
household_rosterandindividual_education, which share the source file and had the identical brokenpidblock.Before → after (cold builds,
LSMS_NO_CACHE=1)household_roster2014-15 rows.dtaindividual_education2014-15 rowss02q23On the uniqueness proof — the caveat matters. The returned frame tests
is_unique == Trueboth before and after. That is the disease, not the cure: the collapse is what makes it unique. The load-bearing check is uniqueness of the declared key in the source:(grappe, menage, s01q)= 5,149 groups over 37,175 rows — broken(grappe, menage, s01q00)= 37,175 / 37,175 — exactSpot-checks against the raw
.dta: HH(1,3)— 9 members, ages[0,1,2,3,5,7,20,28,43], 6M/3F,pid1003001..1003009. HH(1,6)— all 15 members, identical age vector.household_characteristicspicks it up for free (log HSize= ln(9), ln(15)).Scope swept
Every Mali (wave × table) declaring
pidwas checked. 2014-15 is the only affected wave. 2017-18 (codeid) and 2018-19 / 2021-22 (numind) were already unique in source — row counts byte-identical before and after.Worth a maintainer's eye
Mali/2014-15/_/people_last7days.py:48already buildspidas(grappe, menage, s01q00)and asserts(t,i,pid)uniqueness.data_info.ymlhad simply drifted out of that keyspace. The repo already contained the evidence that the wiring was wrong.s01qthat varies across member rows — different respondents answered for different members. Harmless now (pidno longer uses it and it reaches no table), but if respondent identity is ever wanted, it belongs inmyvars, never as an index level.interview_dateis deliberately untouched — it needs the{const: value}idxvars primitive and ships separately (per D3).Tests: 231 passed. Ledger:
.coder/ledger/323-mali-pid.md.Refs #323.
🤖 Generated with Claude Code