fix(#323/#627): Malawi's and Guinea-Bissau's cartesian dfs: merges, and #637's three Malawi .first() sites

.coder/ledger/323-malawi-gb-cartesian.md

@@ -0,0 +1,153 @@
+# Prior-Art Ledger — GH #323 site 4 / #627 (Malawi + Guinea-Bissau cartesians) and #637 (Malawi `.first()` sites)
+
+> Per-task ledger. Inherits the repo §0 baseline in `STANDING.md`; cites
+> `CLAUDE.md` and `lsms_library/data_info.yml` rather than re-copying them.
+
+**Search tier used:** ripgrep + git floor. gitnexus not consulted — the config
+half touches no symbol, and the `malawi.py` half is a comment + one `dropna`
+inside a function whose only callers are the four wave scripts that import it
+by name (enumerated by grep).
+
+## §1 Task, restated
+
+Three of the eight cartesian `dfs:` cells from PR #627's 40-country census were
+unowned after Mali's went in PR #641:
+
+| cell | L x R | merged | phantom |
+|---|---|---|---|
+| Malawi 2010-11 | 12,271 x 12,271 | 196,083 | 183,812 |
+| Malawi 2019-20 | 14,612 x 11,434 | 185,842 | 171,230 |
+| Guinea-Bissau 2018-19 | 5,351 x 450 | 5,410 | 59 |
+
+Plus: review the three `groupby().first()` sites in `Malawi/_/malawi.py` for
+KEY SOUNDNESS (#637). Config/script only, no `aggregation:` key, fix the merge
+never the aftermath.
+
+## §2 Existing machinery (this task's area)
+
+| symbol | path | what it does | tested? | reuse / extend / new |
+|---|---|---|---|---|
+| `Wave._cartesian_keys` / `_merge_subframes` | `origin/fix/323-site4-dfs-merge` `country.py:909/945` | exact many-to-many detector + phantom count | 16 tests on that branch | **measuring instrument only**; core untouched |
+| `Wave.grab_data` `dfs:` block | `country.py:~1112-1215` | outer-merges sub-frames on `merge_on ∪ {t}` | yes | untouched |
+| sub-frame `df_edit` dispatch | `country.py:802` + `:997` — `column_mapping(<sub-frame name>, ...)` then `mapping_info.pop('df_edit')` | runs a country/wave-module function named after the SUB-FRAME on that sub-frame, **before** the merge | no direct test before this PR | **used** for Guinea-Bissau; now pinned by a test |
+| `_collapse_to_cluster_grain` (site 2) | `country.py:~4490` | projects household-grain `cluster_features` onto `(t, v)` | yes | not touched — it is why `Wave.cluster_features()` is blind here |
+| PR #641 `Mali/2021-22` | `Mali/2021-22/_/data_info.yml` | deleted the `dfs:` block for a single-file extraction | yes | **template**; not applicable to either country here (see §5) |
+| PR #639 `fix/323-malawi-config` | `Malawi/{2004-05,2016-17}/_/data_info.yml` | fixed 2004-05's cluster key and 2016-17's `df_geo` merge key | yes | **this branch is stacked on it**; its two "deliberately not fixed" comments are replaced |
+| `tests/conftest.py::requires_s3` | `tests/conftest.py:71` | data-free-CI skip marker | yes | reused |
+
+## §3 Definitions & conventions in force
+
+- **D1, fix the merge, never reduce afterwards** — `CLAUDE.md` "Grain Collapse";
+  `SkunkWorks/grain_aggregation_policy.org` §3a. `aggregation:` is dead config.
+- **`dfs:` merges exist to be collapsed, not grown** — `CLAUDE.md` "Gotchas with
+  Teeth": *"Existing `dfs:` merges are grandfathered but should be collapsed
+  when touched"*.
+- **The published cluster GPS is a displaced cluster fix stamped on every
+  household**, not household GPS — `CLAUDE.md` site-2 note (GH #161). Verified
+  here for both Malawi waves (0 of 768 / 0 of 717 EAs carry two coordinates).
+- **`.first()` skips NA per column**, so a conflicting group yields a composite
+  belonging to no real row — `CLAUDE.md` "Grain Collapse". Wrong only when the
+  duplicate rows are DIFFERENT ENTITIES; `skipna=False` was drafted and
+  abandoned repo-wide.
+- **EHCVM cluster identity**: `v: grappe`, `i: [grappe, menage]` — `CLAUDE.md`
+  "EHCVM countries".
+- **Only 2016-17's cross-sectional half is `cs-17-` prefixed in the wave
+  scripts; every other half emits the raw wave hhid and relies on `id_walk` /
+  `panel_ids` chaining** — `Malawi/_/CONTENTS.org`, "plot_features (GH #167)".
+  Read *before* concluding anything about 2019-20's `i_prefix=''`; it is the
+  documented decision, not a defect (see §4, trap 3).
+- **IHS4's GPS displacement is "nearly but not exactly EA-constant"** — 7 of
+  779 EAs carry more than one `lat_modified` — `Malawi/_/CONTENTS.org`,
+  "GH #323: 2016-17 had no GPS". So the EA-constant-coordinate invariant this
+  PR asserts holds for 2010-11 and 2019-20 and is *documented not to* for
+  2016-17; the test is parametrized accordingly and says why.
+
+## §4 Invariants & assumptions
+
+- **A merge key duplicated in BOTH sub-frames is a cartesian by construction** —
+  `_cartesian_keys` docstring; sound *and* complete.
+- **The warm cache hides all of this.** L2-country is written post-collapse.
+  Every number here was measured cold, in an isolated `LSMS_DATA_DIR` with only
+  `dvc-cache` symlinked, with `LSMS_NO_CACHE=1`, against #627's core on
+  `PYTHONPATH` (asserted in-process, not assumed).
+- **`Wave.cluster_features()` cannot see a cartesian for Malawi** — site 2
+  projects to `(t, v)` first, giving 768 / 819 rows with or without the bug.
+  Tests must call `grab_data('cluster_features')`. (Guinea-Bissau declares no
+  `i`, so site 2 does not fire and its wave frame is the merged frame.)
+- **An in-process `.first()` patch cannot see a `materialize: make` build** —
+  hence `runpy` (PR #646's method). All three #637 tables are `materialize:
+  make`.
+- **"Exact duplicates" is not reassurance and invariance can be missingness** —
+  PR #646 / the Tanzania key. Disposed of here by there being no duplicate
+  groups at all on a non-null key, which is a stronger statement than "the
+  duplicates agreed".
+- **A BROKEN KEY CAN PRODUCE ZERO DUPLICATES** (#637 trap 3, the Tanzania
+  `shocks` inversion). A key from the wrong namespace makes every replicated
+  row its own distinct "household", so the duplicate-count instrument reads
+  clean on a thoroughly broken key. **"0 duplicates" is therefore necessary,
+  not sufficient**; the discriminating check is a **per-wave** overlap of the
+  table's `i` against the roster's — per wave, because clean waves otherwise
+  carry broken ones through an aggregate check. Run for all three Malawi
+  sites: **100% in every wave**, matching literally (`101011000014` on both
+  sides), with no row-count inflation relative to source households ×
+  strategies. Worth running rather than waving away: Malawi 2019-20 has the
+  surface shape of a namespace split — `cs_i` (`'cs-19-' + format_id`) in
+  `data_info.yml` for `sample` / `household_roster` / `cluster_features` vs.
+  `i_prefix=''` in all three wave scripts. **`CONTENTS.org` already records
+  that asymmetry as deliberate** (§3 above), and the measurement confirms it:
+  the prefix survives to the API on neither side (0 of 14,612 ids carry it, in
+  any table), so both are in the raw `case_id` namespace and they agree.
+
+## §5 Reuse decision
+
+| quantity | decision | reason |
+|---|---|---|
+| Malawi `df_geo` key | **re-key** `v: ea_id` → `i: case_id` (`cs_i` in 2019-20), `merge_on: [i]` | files are exactly 1:1 on `case_id`; #627's first named remedy |
+| Malawi: delete the `dfs:` block (Mali's cure) | **rejected** | lat/lon exist only in the geovariables file; the cover page has none |
+| Guinea-Bissau: re-key on `i` | **rejected** | the GPS file is genuinely grappe-grain and carries nothing finer; both copies of every duplicate agree on `vague` |
+| Guinea-Bissau: single-file cure | **rejected** | cover page has no GPS columns at all (checked) |
+| Guinea-Bissau: `df_geo` hook + `drop_duplicates()` | **taken** | #627's *second* named remedy — reduce the sub-frame to merge-key grain BEFORE the merge. Lossless: the 5 extra rows are byte-identical incl. GPS timestamp |
+| Guinea-Bissau: `groupby().first()` / `keep='first'` | **rejected** | would silently choose between two disagreeing fixes if one ever ships; `drop_duplicates()` lets the guard fire instead |
+| Guinea-Bissau: convert to a `materialize: make` script | **rejected** | far heavier than 59 rows warrant, and moves a YAML-expressible table onto the script path against the repo's stated preference |
+| `.first(skipna=False)` at any #637 site | **rejected** | no multi-row groups exist; the repo abandoned that approach |
+| `aggregation:` key anywhere | **rejected** | dead config, D1 |
+
+## §6 Open questions for the human
+
+- **The Guinea-Bissau hook is dispatched BY NAME off the sub-frame key.** This
+  is real, current behaviour on both `development` and #627's branch, but it is
+  undocumented and there was **no prior art for it** (`rg '^def df_geo'` across
+  all 40 countries: zero hits). It is heavily documented in three places and
+  pinned by a test, but a reviewer may prefer a different mechanism. If so, the
+  alternatives are ranked in §5.
+- **2019-20 at HOUSEHOLD grain**: 8 IHS5 geo rows have a NULL coordinate while
+  their 15 EA siblings carry the EA's fix. Under the `v`-merge those 8
+  households borrowed a sibling's value; under the `i`-merge they get their own
+  NULL. The returned `(t, v)` table is unchanged (`.first()` skips NA), and the
+  household-grain intermediate is now honest — but it *is* a behavioural
+  difference, and it is the only one.
+- **`food_coping` / `months_food_inadequate` build only the cross-sectional
+  half in 2016-17 and 2019-20** (12,447 and 11,434 households; the IHPS panel
+  halves are absent). Observed while auditing #637; a coverage gap, not a key
+  defect, and out of scope here.
+
+---
+### Phase 3 — verification
+
+- `Malawi/{2010-11,2019-20}/_/data_info.yml` — **OK (§3, §5)**: `merge_on: [i]`,
+  no `aggregation:`, cured at the merge. Phantom 183,812 → 0 and 171,230 → 0.
+- `Guinea-Bissau/2018-19/_/{data_info.yml,mapping.py}` — **OK (§3, §5)**:
+  reduced before the merge, de-duplication not aggregation. Phantom 59 → 0.
+- Value preservation — **OK (§4)**: country-level `cluster_features`
+  bit-for-bit identical for both countries (`DataFrame.equals` True, index
+  equal, dtypes equal; Malawi 3,235 x 5, Guinea-Bissau 450 x 4).
+- `tests/test_gh323_malawi_gb_cartesian.py` — **OK (§4)**: asserts on
+  `grab_data`, not `Wave.cluster_features()`. **Negative control run** (pre-fix
+  configs restored, fresh isolated data root): `6 failed, 10 passed` — the three
+  cartesian tests report 5410 / 196083 / 185842 and the three structural tests
+  fail, while the 10 source-invariant and cluster-count tests pass *with the bug
+  fully present*, which is exactly the blindness the module docstring warns
+  about. After: `16 passed`.
+- `Malawi/_/malawi.py` — **OK (§3)**: no `skipna=False`, no reducer added; one
+  `dropna(subset=['plot_id'])` that makes an existing silent deletion explicit
+  and is a provable no-op (`groupby(dropna=True)` already removed those rows).

lsms_library/countries/Guinea-Bissau/2018-19/_/data_info.yml

@@ -56,6 +56,31 @@ cluster_features:
                     urbain: Urban
                     URBAIN: Urban
     df_geo:
+        # GH #323 site 4 / GH #627 -- CARTESIAN, CURED BEFORE THE MERGE.
+        #
+        # `grappe_gps_gnb2018.dta` ships 450 rows for 445 grappes: five
+        # grappes appear TWICE as VERBATIM duplicate records (identical lat,
+        # lon, accuracy, altitude and GPS timestamp).  df_main is the
+        # household cover page at ~12 rows per grappe, so merging on `v` when
+        # `v` repeats on BOTH sides paired 11-12 household rows with 2 GPS
+        # rows for each of those five: 5,351 -> 5,410 rows, 59 phantom.
+        #
+        # Guinea-Bissau is NOT the Malawi/Mali shape.  There is no household
+        # id to re-key on (the GPS file is genuinely at grappe grain and
+        # carries only `grappe` + `vague`), and no single-file cure (the
+        # cover page has no GPS columns at all -- checked).  So this cell
+        # takes the OTHER remedy #627's guard names: reduce the sub-frame to
+        # the merge-key grain BEFORE the merge.  The `df_geo` hook in
+        # ../_/mapping.py does it with drop_duplicates() over every column,
+        # which removes only rows identical to one already present -- so two
+        # GPS fixes that actually DISAGREE would both survive and re-trip the
+        # guard rather than being silently reduced to one.  READ THAT
+        # DOCSTRING: the hook is dispatched by matching this sub-frame's key
+        # name, which is not obvious from either file alone.
+        #
+        # Not fixed here: 5 of the 450 grappes on the cover page have no GPS
+        # record at all, so they carry NaN coordinates.  Absent data, not
+        # manufactured data.
         file: grappe_gps_gnb2018.dta
         idxvars:
             v: grappe

lsms_library/countries/Guinea-Bissau/2018-19/_/mapping.py

@@ -5,6 +5,61 @@
 from lsms_library.transformations import food_acquired_to_canonical as _food_acquired_canonical
 
 
+def df_geo(df):
+    """``cluster_features.df_geo`` sub-frame hook — GH #323 site 4 / GH #627.
+
+    **This is a SUB-FRAME hook, not a table hook.**  ``Wave.grab_data``
+    resolves a ``dfs:`` sub-frame's ``df_edit`` by calling
+    ``column_mapping(<sub-frame name>, ...)``, so a module-level function
+    named exactly after the sub-frame key (``df_geo``) runs on that
+    sub-frame **after extraction and before the merge**.  That "before the
+    merge" is the whole point; see below.
+
+    ``grappe_gps_gnb2018.dta`` ships **450 rows for 445 grappes**: five
+    grappes (155325, 155426, 199719, 199816, 1771231) appear TWICE, as
+    VERBATIM duplicate records — same latitude, same longitude, same
+    accuracy, same altitude, same GPS timestamp.  The companion community
+    cover ``s00a_co_gnb2018.dta`` carries the identical 450/445 split, so
+    the duplication happened once, on the community side of the export.
+
+    That alone made the ``cluster_features`` merge a CARTESIAN PRODUCT.
+    The other sub-frame (``s00_me_gnb2018.dta``, the household cover page)
+    is HOUSEHOLD grain — ~12 rows per grappe — so for those five grappes
+    the merge on ``v`` paired 11-12 household rows with 2 GPS rows and
+    emitted 22-24 where an honest join emits 11-12: **5,351 -> 5,410 rows,
+    59 of them phantom** (PR #627's 40-country census).
+
+    The cure is the second remedy the #627 guard itself names — *"reduce a
+    sub-frame to the merge-key grain BEFORE it is merged"*.  It is
+    deliberately NOT the first one (*"merge on a key unique in at least one
+    sub-frame"*): Guinea-Bissau has no such key.  The GPS file is at grappe
+    grain and carries nothing finer (only ``grappe`` and ``vague``, and both
+    copies of every duplicate agree on ``vague``), while the cover page is
+    at household grain, so ``v`` is non-unique on *both* sides and no
+    declarable key separates the two copies.
+
+    ``drop_duplicates()`` over EVERY column — key and payload alike — is
+    used on purpose, and it is de-duplication, not aggregation (#323 D1
+    forbids the latter).  It removes a row only when an identical row
+    already exists, so it cannot choose between disagreeing records: if a
+    future export ever ships two GPS fixes for one grappe that actually
+    DIFFER, both survive, the merge is cartesian again, and #627's guard
+    fires.  A ``groupby().first()`` or ``keep='first'`` would silently pick
+    one and hide exactly that.
+
+    Unrelated to the cardinality and NOT fixed here: the household cover
+    page names 450 grappes but the GPS file only 445, so five grappes carry
+    NaN coordinates.  Missing data, not manufactured data — and note the
+    two counts coincide only by arithmetic accident: the five duplicated
+    records are byte-identical, so they are not five mislabelled fixes that
+    could be reassigned to the five grappes that lack one.
+    """
+    names = list(df.index.names)
+    flat = df.reset_index()
+    deduped = flat.drop_duplicates()
+    return deduped.set_index(names)
+
+
 # Lossy-substitution prefixes the source data carries for two specific
 # Portuguese words.  Upstream of pyreadstat, the export pipeline
 # replaced the second byte of certain UTF-8 codepoints with literal

lsms_library/countries/Guinea-Bissau/_/CONTENTS.org

@@ -13,6 +13,57 @@ each grappe is visited in exactly one =vague=, so household
 identifier is =(grappe, menage)= without a vague level.  See
 CLAUDE.md "EHCVM countries" for the cross-country pattern.
 
+** =grappe_gps_gnb2018.dta= ships five duplicated records (GH #323 site 4 / #627)
+
+The GPS file has *450 rows for 445 grappes*.  Five grappes -- 155325,
+155426, 199719, 199816 and 1771231 -- appear *twice*, as verbatim
+duplicate rows: same latitude, same longitude, same accuracy, same
+altitude, same GPS timestamp.  The companion community cover
+=s00a_co_gnb2018.dta= carries the identical 450/445 split, so the
+duplication happened once, on the community side of the export.
+
+That alone made =cluster_features= a *cartesian product*.  The other
+=dfs:= sub-frame is the household cover page =s00_me_gnb2018.dta= at
+~12 rows per grappe, so merging on =v= when =v= repeats on *both*
+sides paired 11-12 household rows against 2 GPS rows for each of those
+five: *5,351 -> 5,410 rows, 59 phantom* (PR #627's census).  Small, but
+the same disease as Malawi's 183,812.
+
+*Guinea-Bissau is not the Malawi shape and does not take the Malawi
+cure.*  There is no household id to re-key on -- the GPS file is
+genuinely at grappe grain and carries nothing finer than =grappe= +
+=vague=, and both copies of every duplicate agree on =vague=.  Nor is
+there Mali 2021-22's single-file cure: the cover page has *no GPS
+columns at all* (checked).  So this cell takes the other remedy #627's
+guard names -- *reduce the sub-frame to the merge-key grain before it
+is merged* -- via a =df_geo(df)= hook in =2018-19/_/mapping.py=.
+
+Two things about that hook are worth knowing before touching it:
+
+- It is dispatched *by name off the sub-frame key*.  =Wave.grab_data=
+  resolves a =dfs:= sub-frame's =df_edit= with
+  =column_mapping(<sub-frame name>, ...)=, so the function must be
+  called exactly =df_geo= and the sub-frame must stay keyed =df_geo=.
+  Renaming either silently drops the de-dup.  A test pins both.
+- It uses =drop_duplicates()= over *every* column, key and payload
+  alike.  That is de-duplication, not aggregation (#323 D1 forbids the
+  latter): a row is removed only when an identical row already exists,
+  so if a future export ever ships two GPS fixes for one grappe that
+  actually *differ*, both survive, the merge is cartesian again, and
+  #627's guard fires.  =groupby().first()= or =keep='first'= would
+  silently pick one and hide exactly that.
+
+After: 5,351 wave rows, *0 phantom*, and the returned 450-row
+=cluster_features= frame is bit-for-bit identical (=DataFrame.equals=
+True, index equal, dtypes equal).
+
+Separately, and *not* fixed: the cover page names 450 grappes but the
+GPS file only 445, so five grappes carry NaN coordinates.  Absent data,
+not manufactured data.  The two counts coincide only by arithmetic
+accident -- the five duplicated records are byte-identical, so they are
+*not* five mislabelled fixes that could be reassigned to the five
+grappes that lack one.
+
 * Household Presence / MonthsSpent
 
 The EHCVM questionnaire (used in the 2018-19 wave) does NOT ask