fix(#323/#627): Malawi's and Guinea-Bissau's cartesian dfs: merges, and #637's three Malawi .first() sites

[ligon]

Stacked on #639 (fix/323-malawi-config), which this branch is based on —
its commit appears in the diff below and should merge first. Everything after
that commit is this PR.

Closes the last two Malawi cells and the Guinea-Bissau cell from #627's
cartesian census, and answers #637 for Malawi's three groupby().first()
sites.

Piece 1 + 3 — the cartesians

All measured cold, in an isolated LSMS_DATA_DIR with only dvc-cache
symlinked, LSMS_NO_CACHE=1, and #627's _cartesian_keys detector on
PYTHONPATH (asserted in-process, not assumed).

cell	wave rows before	after	phantom before	after
Malawi 2010-11	196,083	12,271	183,812	0
Malawi 2019-20	185,842	14,612	171,230	0
Guinea-Bissau 2018-19	5,410	5,351	59	0

Values do not move. Country-level cluster_features is bit-for-bit
identical before and after for both countries — DataFrame.equals True, index
equal, dtypes equal (Malawi 3,235 × 5 over all five waves; Guinea-Bissau
450 × 4).

Malawi — a household-grain frame keyed on the cluster

Both sub-frames are one row per household, merged on v, so pd.merge paired
every household of an EA with every other; _normalize_dataframe_index then
collapsed the wreck to 768 / 819 clusters, so the table read clean everywhere
anyone looked. The geo files carry case_id, so the honest key was always
there: idxvars: {i: case_id} + merge_on: [i] (2019-20 through cs_i, as its
df_main does).

What I verified vs. what I took from #639. #639's author implemented this
same fix, measured it, then deliberately reverted it so #627 would keep its
evidence, leaving the diagnosis in the YAML. I re-derived every load-bearing
claim from source rather than trusting the note, and all of it holds:

• the file pairs are exactly 1:1 on case_id — 12,271 = 12,271 and
  11,434 = 11,434, zero rows on either side alone, no duplicate ids;
• lat/lon are constant within all 768 and all 717 EAs (0 EAs with two
  coordinates) — the published GPS is the EA's displaced fix stamped on each
  household, so a household's own row is what the v-merge was giving it;
• df_main.ea_id == df_geo.ea_id for every shared household;
• the 102 IHPS panel EAs share no ea_id and no household id with the
  IHS5 geo file, so their NaN coordinates are absent data, unchanged.

One honest wrinkle, at household grain only: 8 IHS5 geo rows carry a NULL
coordinate while their 15 EA siblings carry the EA's fix. Under the v-merge
those 8 households borrowed a sibling's value; under the i-merge they get
their own NULL. The returned (t, v) table is unchanged (.first() skips NA),
and the household-grain intermediate now tells the truth about which rows the
survey actually geolocated.

Guinea-Bissau — a different shape, and a different cure

Not two household-grain frames. grappe_gps_gnb2018.dta ships 450 rows for
445 grappes: five appear twice as verbatim duplicate records — same lat,
lon, accuracy, altitude, GPS timestamp. (The companion community cover
s00a_co_gnb2018.dta carries the identical 450/445 split, so the duplication
happened once, on the community side of the export.)

Neither of the other cures applies: there is no household id to re-key on (the
GPS file is genuinely grappe-grain and carries nothing finer than grappe +
vague, and both copies of every duplicate agree on vague), and no
single-file cure à la Mali (the cover page has no GPS columns at all —
checked). So this takes #627's other named remedy — reduce the sub-frame to
the merge-key grain before it is merged — via a df_geo hook in the wave's
mapping.py.

drop_duplicates() over every column, key and payload alike: that is
de-duplication, not aggregation, and it removes a row only when an identical row
already exists — so two GPS fixes that actually disagree would both survive
and re-trip #627's guard rather than being silently reduced to one.
groupby().first() / keep='first' would hide exactly that.

⚠️ Reviewer, look here. The hook is dispatched by name off the sub-frame
key — Wave.grab_data resolves a dfs: sub-frame's df_edit via
column_mapping(<sub-frame name>, ...). That is real, current behaviour on
both development and #627's branch, but it is undocumented and there is no
prior art for it anywhere in the corpus (rg '^def df_geo' over all 40
countries: zero hits). It is documented in the YAML, the docstring and
CONTENTS.org, and pinned by a test. If you'd rather this site grew an
explicit declarative lever instead, say so — ranked alternatives are in the
ledger §5.

Piece 2 — #637's three .first() sites in Malawi/_/malawi.py

Method: patched DataFrameGroupBy.first in-process and drove the wave
scripts with runpy (PR #646's method). All three tables are
materialize: make, so an in-process patch cannot see them through the
subprocess — the mistake that made an earlier probe of a site like this report a
meaningless "0".

site	groups > 1 row	verdict
plot_features_for_wave	0 (on a non-null key)	key sound; .first() is a no-op
food_coping_for_wave	0	key sound; .first() is a no-op
months_food_inadequate_for_wave	0	key sound; .first() is a no-op

No composite is ever built at any of the three, because no group ever has
more than one row to build one from. For the two Module-H sites rows in equals
groups out exactly — 61,355 / 19,994 / 62,232 / 57,157 and 12,271 / 4,000 /
12,447 / 11,434, the latter being precisely each wave's Module H household
count.

The three traps, and which check answers each

"They're only exact duplicates" is not reassurance, and a key can look
invariant by being null. Both are disposed of by the exact equality above:
there are no duplicate groups at all, which is strictly stronger than "the
duplicates agreed"; and the household id is non-null in every row of all three
tables. A household id shared by two rows, or two distinct ids merged by
format_id, would each surface right there as a duplicate group. Confirmed at
source: case_id / y2_hhid are unique in Module H and format_id collapses
none of them.

A broken key can produce zero duplicates — the Tanzania inversion, which
turns this instrument inside out. "0 duplicates" is necessary, not
sufficient, so I ran the discriminating check separately: a per-wave
overlap of each table's i against the roster's.

table	2010-11	2013-14	2016-17	2019-20
plot_features	100%	100%	100%	100%
food_coping	100%	100%	100%	100%
months_food_inadequate	100%	100%	100%	100%

Per wave, never in aggregate. And not vacuously — the id strings match
literally (101011000014 on both sides), and the second signal agrees: row
counts show no namespace inflation (never above source households ×
strategies).

This one was worth running rather than waving away. Malawi 2019-20 has the
surface shape of a namespace split: its data_info.yml runs case_id
through cs_i ('cs-19-' + format_id) for sample / household_roster /
cluster_features, while all three wave scripts pass i_prefix='' / bare
format_id.

Malawi/_/CONTENTS.org already records that asymmetry as a deliberate
decision, and I am citing it rather than re-deriving it — "2016-17
cross-sectional id is cs-17--prefixed in sample() …; the wave script applies
the same prefix so XS plots are not ~100% orphaned. All other halves emit the
raw wave hhid and rely on framework id_walk / panel_ids chaining." 2019-20 is
one of those other halves. The measurement confirms the documented
expectation: the cs-19- prefix survives to the API on neither side (0 of
14,612 ids carry it, in any table), so both are in the raw case_id namespace
and i joins exactly. 2016-17 reaches the same place by the other route.
test_i_lives_in_the_same_namespace_as_the_roster pins both per wave, with
household counts, so neither route can drift.

plot_features showed 4 apparent duplicate groups (3 in 2016-17 CS, 1 in its
Panel half); every one consists entirely of rows whose plot_id is NULL.

No .first(skipna=False). It would change nothing here, and the repo has
already rejected that approach.

What the audit did find: 468 rows deleted on a null key

groupby() defaults to dropna=True, so a plot_features row with a NULL
plot_id was being deleted without a word — #323 §3b's delete-and-report, in
a country script rather than in core:

2010-11	2013-14	2016-17	2019-20	total
275	158	27 + 6	0 + 2	468

They are empty Module C roster stubs, not plots: 0 of all 468 carries a
non-null value in any of ag_c04a / ag_c04b / ag_c04c. A row with no plot
id and no measurement cannot go into a (t, i, plot_id) table, so dropping it is
right — it just should not have been silent. It is now an explicit
dropna(subset=['plot_id']) immediately before the groupby, which is a provable
no-op: the three built tables are bit-for-bit identical before and after.

Tests, and the negative control

tests/test_gh323_malawi_gb_cartesian.py (16 tests) asserts on
grab_data('cluster_features'), not Wave.cluster_features() — the latter
runs the site-2 projection onto (t, v) first and returns 768 / 819 rows for
Malawi with or without the cartesian. My first draft asserted on the wrong one
and passed with the bug present.

Negative control (pre-fix configs restored, fresh isolated data root):
6 failed, 10 passed. The three cartesian tests report 5410 / 196083 / 185842;
the three structural tests fail; and the ten source-invariant and cluster-count
tests pass with the bug fully present — exactly the blindness the module
docstring warns about, and why the row-count assertions are where they are.
After the fix: 16 passed.

tests/test_gh637_malawi_first_sites.py pins the invariants the #637 verdicts
rest on. It has no meaningful negative control, and I want to be straight about
that: #637 produced no behavioural change, so there is no "before" state for
these tests to fail against. They are forward-looking regression pins.

Both files use requires_s3 so they skip cleanly in the data-free CI job.

Read against the country documentation

Malawi/_/CONTENTS.org (757 lines) and Guinea-Bissau/_/CONTENTS.org were
read end to end before concluding. Three things came out of that:

1. The 2019-20 id-prefix asymmetry is already documented as deliberate —
   cited above rather than re-derived. Reading it first is what kept "0
   duplicates on a bare case_id" from being written up as a namespace defect.
2. IHS4's GPS displacement is documented as "nearly but not exactly
   EA-constant" (7 of 779 EAs). The new EA-constant-coordinate test is
   therefore parametrized over 2010-11 and 2019-20 only — where I measured 0 of
   768 and 0 of 717 — and its docstring says explicitly not to widen it to
   2016-17, citing the passage.
3. One arithmetic error found and fixed: the Sampling Design overview read
   "IHS5 (2019-20): cross-sectional (11,434) + panel (3,178) = 14,552
   households". The addends are right; the total is not. 14,612 is what
   sample(), household_roster() and cluster_features.df_main all return,
   and it is the figure this PR's row counts rest on.

Guinea-Bissau's file had nothing recorded about the GPS file or the cluster
merge, so the duplicated-records finding is new and has been added there.

Scope

Config/script only. Nothing under lsms_library/*.py outside
countries/Malawi/** and countries/Guinea-Bissau/**. No aggregation: key,
no declared reducer, no dvc CLI invocation. Ledger:
.coder/ledger/323-malawi-gb-cartesian.md.

Refs #323, #627, #637.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr