fix(household_characteristics): residence filter was silently deleting 8 entire wave-cells#594
Merged
Merged
Conversation
…teristics is the only site; BF-2014 French labels are a config fix (reuse the EHCVM mapping: idiom), the per-t all-NaN guard is the one 'new' piece Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…ole waves
`roster_to_characteristics()` is handed the *country-level* concat of every
wave's `household_roster`, so a residence-duration column contributed by one
wave is unioned (all-NaN) onto every other wave. Three defects followed, each
silently returning NOTHING for a wave whose roster is fully populated:
D1 The resolution was an if/elif chain over the columns *present*, not
*populated*. Ethiopia carries both MonthsAway (W1-W3) and WeeksAway
(W4-W5, when the questionnaire switched to weeks -- exactly as CLAUDE.md
documents). MonthsAway won the elif, resolved to all-NaN on W4-W5, and
both waves vanished. -> per-ROW coalesce across the three sources,
priority MonthsSpent > MonthsAway > WeeksAway.
D2 No all-NaN guard. CotedIvoire 1985-89 and Mali 2021-22 never asked a
residence question; their MonthsSpent column exists only because a later
wave supplies it. The keep-mask was all-False -> 5 waves vanished.
-> per-`t` fallback to the documented "count everyone" behaviour ("countries
without any residence column are unaffected", CLAUDE.md), applied at wave
granularity, and it now warns rather than deleting silently.
D3 Burkina Faso 2014's MonthsSpent (EMC B3A) is the raw French 6-month binary
'6 mois ou plus' / 'moins de 6 mois'; to_numeric(errors='coerce') NaN'd the
wave away. -> fixed in CONFIG (the wave's data_info.yml `mapping:`),
following the EHCVM Oui/Non -> 12/0 pattern. No French in library code.
The filter itself is untouched wherever it has data: the fallback fires only
for a wave with no usable residence datum at all. Census over all 34 countries
that expose household_characteristics: the 8 dead wave-cells come back (+34,974
households) and all 88 other waves are byte-identical, Uganda included -- so the
1315-HH Uganda drift the filter was introduced to fix stays fixed.
Ledger: .coder/ledger/roster-characteristics-wave-deletion.md
Tests: tests/test_roster_residence_filter.py (6 of them fail pre-fix)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
ligon
added a commit
that referenced
this pull request
Jul 12, 2026
…s wrong Addendum #3 claimed "`PYTHONPATH` alone does NOT redirect imports of `lsms_library` to a worktree". That is FALSE -- PYTHONPATH does beat the `.pth`. The wrong claim sent agents to a needless mitigation (build a fresh venv inside the worktree, or verify by static diff only). The ACTUAL trap is `python <script>.py`. Python sets `sys.path[0]` to the SCRIPT's own directory, not cwd -- so `cd`-ing into the worktree does not protect a script run: cwd never enters `sys.path` at all, the `.pth`'s main-repo path wins, and you import the MAIN checkout while believing you are testing the worktree. Verified (cwd = worktree throughout): python -c "import lsms_library" PYTHONPATH unset -> worktree (cwd is sys.path[0]) python bench/scan.py PYTHONPATH unset -> MAIN CHECKOUT <-- the trap python bench/scan.py PYTHONPATH=<wt> -> worktree This is nastier than the old text described, because the agent has done everything that LOOKS right (cd'd into the worktree, run the worktree's own script) and still tested the wrong code -- silently. It has already cost a worker agent a full wasted audit run (found while reviewing PR #594). Since the failure is silent and looks like success, the guidance now says to ASSERT it rather than assume it: import lsms_library assert 'worktrees' in lsms_library.__file__, lsms_library.__file__ Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
Decision on the BF-2014
|
Ethan's call: keep 12/0. The reasoning is not obvious and someone will
otherwise "fix" it later, so it now lives above the mapping itself and in the
ledger (§6, open question -> resolution):
- B3A's variable label is a DURATION question ("durant combien de mois [NOM] a
vecu dans le menage"), but the released variable is collapsed to a 6-month
binary: {1: '6 mois ou plus', 2: 'moins de 6 mois', 9: 'Valeur manquante'},
distributed 77,268 / 943 / (332 NaN).
- So 'moins de 6 mois' does NOT mean "lived zero months" -- those 943 people
(1.2%) DID live in the household, for 1-5 months. Read literally, mapping
them to 0 is false.
- That is where B3A differs from the EHCVM precedent it borrows: s01q12
("lived continuously 6+ months? Oui/Non") is a MEMBERSHIP test, where Non->0
is faithful. B3A is a binned duration.
- We keep 12/0 anyway because the 0 encodes "not a de-facto household member"
(the standard LSMS 6-month rule), not a claim about months lived -- and
consistency with the EHCVM family wins.
- 12/6 was rejected partly because 6 is the one value the respondent explicitly
ruled out. If we ever keep these members, the honest fill is 3 (the 1-5
month interval midpoint), not 6.
Comment-only + ledger; no behaviour change (tests/test_roster_residence_filter.py
still 13/13, including the mapping assertion it reads back out of the YAML).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug:
household_characteristicssilently deletes entire wavesCountry.household_characteristics()is derived at runtime (_ROSTER_DERIVED), androster_to_characteristics()is handed the country-level concat of every wave'shousehold_roster. So a residence-duration column contributed by one wave is unioned(all-NaN) onto every other wave. Where the resolved months series is unusable for a wave,
the keep-mask is all-False for that whole wave and the table returns nothing for it —
while
household_rosteris fully populated. 8 wave-cells were destroyed.Three distinct defects:
if/elifchain picked the first column present, not populated. Ethiopia's country frame carries bothMonthsAway(0 non-null in W4/W5) andWeeksAway(28,634 / 20,504 non-null — the questionnaire switched to weeks, exactly asCLAUDE.mddocuments).monthsawaywon theelif→msall-NaN → both waves deleted.MonthsSpent>MonthsAway>WeeksAway)tfallback to the documented "countries without any residence column are unaffected — the old count-everyone behavior continues" (CLAUDE.md), applied at wave granularity, and it now warns instead of deleting silentlyMonthsSpent(EMCB3A) is'6 mois ou plus'(77,268) /'moins de 6 mois'(943).errors='coerce'→ NaN → wave deleted.mapping:block in the wave'sdata_info.yml, following the EHCVMOui/Non→ 12/0 patternCLAUDE.mdalready prescribesThe diagnosis was confirmed empirically before any edit (raw
B3Avalue counts; per-wavenon-null counts of each residence column) — no part of it turned out to be wrong.
The adversarial constraint: the filter must keep working
The residence filter exists because counting everyone produced a 1315-household drift on
Uganda vs. the replication pipeline. The D2 fallback is that old behaviour, so it must fire
only for a wave with no usable residence datum at all. Two independent proofs:
t, not global. A wave with even one non-null residence value isfiltered exactly as before.
household_characteristics, warm cache — per-waverow counts before vs. after: the 8 dead waves come back (+34,974 households); all 88 other
wave-cells are byte-identical. Uganda's 8 waves: unchanged, and the D2 warning does not
fire for it.
Waves that moved (the only 8)
(Sanity: CotedIvoire 1985-86 → 1,588 HH is the known LSMS sample size; Ethiopia ESS4/ESS5 →
6,768 / 4,946 HH match the published wave sizes.)
Attribution — each defect is fixed by its own fix, nothing is masked
Verified per country after the change:
config mapping (D3), not masked by the fallback.
MonthsSpentis now12.0×77,268 /0.0×943 / NaN ×332 — exactly the raw label counts. 1,097 members still dropped (theshort-stay + not-asked members, minus infants).
WeeksAway, and the filter genuinely fires there (884 / 4,872 members dropped).documented no-residence-column behaviour.
Full per-wave census (all 96 wave-cells)
expand
Tests
tests/test_roster_residence_filter.py(new, synthetic rosters — no microdata needed):TestD1PerRowCoalesce— the Ethiopia shape (MonthsAway in w1, WeeksAway in w2, each NaN inthe other): both waves survive, both still filtered;
MonthsSpentwins on a row where twosources are populated.
TestD2AllNaNWaveGuard— an all-NaN wave is not deleted, counts everyone, and thepopulated wave in the same frame is still filtered; the warning names the dead wave only;
works with no
tlevel too.TestFilterStillFiresWhereItShould— the Uganda guarantee: 0-month member dropped, NaNmember dropped, infant (
age < 1) kept, no fallback warning; string'12'/'12.0'months still parse.
TestD3BurkinaFaso2014Config— pins the BF-2014mapping:block so a config regression iscaught.
6 of these fail on the pre-fix code (verified by stashing the fix).
Verification
pytest(full suite, warm, from the worktree): 3567 passed, 154 skipped, 4 xfailed,1 xpassed, 1 failed. The one failure is
test_currency.py::test_feature_ghana_per_wave,which fails identically on the base branch (confirmed by stashing this PR's changes) —
pre-existing, unrelated.
bench/feature_audit/scan.py --features household_characteristics household_roster(32countries): 0 errors, 0 fails. Diffing the check-level findings against the same scan on
the pre-fix code, the only difference is the expected row-count baseline
(454,342 → 478,587); the 3 B-class findings (GH bug: framework silently drops rows on duplicate canonical index (groupby().first()) #323 duplicate collapse, canonical
mlevelabsent) are unchanged and pre-existing.
Ledger
.coder/ledger/roster-characteristics-wave-deletion.md(own commit). §5 marks the BF-2014label mapping as reuse of the EHCVM
mapping:idiom and the per-tguard as the singlenew piece. §6 carries two open questions for the human:
B3Ais a 6-month binary, so 12/0 (EHCVM convention) drops the 943 short-staymembers. If the project would rather keep them, the mapping becomes
12 / 6.behaviour, but those waves are not filter-comparable with their EHCVM siblings.
Notes
CLAUDE.mdedit (out of scope for an agent), but its "MonthsSpent / MonthsAway /WeeksAway" section would benefit from a sentence on the per-row coalesce + per-wave fallback.
transformations.dummies()raises on a single-row frame —v[s].squeeze()returns ascalar and
zip(*[scalar])iterates its characters.CLAUDE.md's scrum-master addendum saysPYTHONPATHcannot redirectlsms_libraryto a worktree. It can — from a neutral cwd,PYTHONPATHbeats the.pth.The real trap is that a script run (e.g.
bench/feature_audit/scan.py) puts thescript's directory on
sys.path[0], so withoutPYTHONPATHit silently imports the maincheckout. That bit this PR's first audit run; it was re-run correctly.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe