fix(#323) Site 4: the dfs: merge MANUFACTURES the duplicates — 4.9M phantom rows (UNBLOCKED: Ethiopia/Niger/Nigeria configs merged)

.claude/skills/add-feature/sample/SKILL.md

@@ -159,6 +159,89 @@ sample:
         - t
 ```
 
+Note `merge_on: i` — the **household** key, unique in both sub-frames. That is
+what makes this merge a join.
+
+### The `dfs:` merge contract (GH #323 site 4)
+
+**`merge_on` must be unique in at least one sub-frame.** If both sub-frames are
+finer-grained than the merge key, the merge is many-to-many and pandas emits a
+*cartesian product within each key group* — it MANUFACTURES rows that exist in
+no survey. The classic error is joining two **household**-grain frames on the
+**cluster** key `v`: Ethiopia's `cluster_features` did exactly this and produced
+65,508 rows in 2013-14 and 57,786 in 2015-16 from tables that have 433 and 432
+clusters. The downstream `groupby().first()` collapse then tidied it away, so
+the table looked perfectly clean.
+
+`Wave._merge_subframes` now checks this exactly (a key value duplicated in
+*both* sub-frames is the definition of the cartesian) and warns with the phantom
+row count; `LSMS_GRAIN_STRICT` makes it fatal. That variable is read through the
+same `_grain_strict()` predicate as sites 1–2, so all three sites agree on what
+counts as "strict" (`1`/`true`/`yes`, case-insensitive) — and turning it on in
+CI is gated on the whole of #323, not just this site. **Fix it by making the
+merge correct — re-key to `i`, or reduce a sub-frame to the merge-key grain in a
+wave script.** Never by aggregating the explosion away afterwards: core does not
+aggregate (`SkunkWorks/grain_aggregation_policy.org`), and a reducer applied to
+a cartesian only puts a signature on the corpse.
+
+#### `merge_how:` (optional, default `outer`)
+
+```yaml
+    merge_on:
+        - i
+    merge_how: left      # default is `outer`
+```
+
+Declare `merge_how: left` when the **primary** sub-df (`dfs[0]`) is authoritative
+for which rows exist and the others are strict enrichments. A geovariable file
+that carries households the cover page does not is the usual case: under `outer`
+those orphans arrive with a null value in every index level the cover owned.
+
+**Be accurate about what this buys you.** Measured on Ethiopia 2013-14 (geo file
+carries 25 households the cover page does not):
+
+| | wave rows | null-`v` rows | delivered clusters | ΣLatitude | `District` dtype |
+|---|---:|---:|---:|---:|---|
+| `outer` | 5,287 | 25 | 433 | 4070.3702 | float64 |
+| `left`  | 5,262 | 0  | 433 | 4070.3702 | int8 |
+
+The orphans do **not** "collapse into one phantom cluster" — the downstream
+cluster-grain collapse *deletes* them, because `groupby` drops null keys. The
+delivered values are identical. So `merge_how: left` is not a data fix; it is
+worth declaring because it stops manufacturing null-keyed rows for site 1 to
+delete, and because it stops the merge widening an integer column. It has a
+cost: under `outer` the site-1 grain report *told* you 25 geo households had no
+cover page. `left` drops them silently. If that signal matters for a wave, keep
+`outer` and let site 1 report it.
+
+#### A dropped sub-df that owned a required column is now a hard error
+
+The GH #515 fallback drops a secondary sub-df that fails to load and proceeds
+with a warning. That is right for a file that is *unavailable* (nothing you can
+edit fixes it). It is **wrong** for a file that loaded fine but does not carry
+the column your YAML names — a typo, a casing mismatch (`lat_dd_mod` vs
+`LAT_DD_MOD`), a renamed variable. Ethiopia lost `Latitude`/`Longitude` from
+three of five waves exactly that way, behind a warning nobody read. If such a
+drop leaves a column declared **required** in `data_scheme.yml` entirely absent,
+`grab_data` now raises. Presence is judged after `derived:`, `drop:` and the
+wave's `df_edit` hook, so a column the wave module supplies is not reported
+absent.
+
+**Fix the column name.** In all ten cells this guard caught, the data existed:
+Ethiopia and Nigeria were casing mismatches or a wrong key column, and Niger
+2011-12's coordinates were in a *sibling* file in the same directory
+(`NER_EA_Offsets.dta`, `LAT_DD_MOD`/`LON_DD_MOD`) while the file the YAML named
+had none.
+
+**Know what this guard is and is not.** It is *"a mis-named column in a `dfs:`
+sub-df is fatal"* — not *"a required column is never absent"*. It fires only
+when a sub-df was dropped, so a wave with no `dfs:` block can serve a declared
+column 100% absent and never trip it (Niger **2014-15** does: `Latitude` is
+declared `float`, 0 of 270 rows populated, no raise). And `optional: true` is a
+blunt escape hatch — `data_scheme.yml` is **country**-grain, so it disarms the
+column for *every* wave and every script-path build of that country while the
+check is per-wave. Use it only where the country genuinely never has the column.
+
 ### Multi-round files (Tanzania 2008-15 pattern)
 
 When a single `.dta` file contains multiple survey rounds with a `round` column, the YAML path cannot handle it --- use a Python script. The script reads the file, maps round numbers to wave labels, and splits panel vs refresh households for `panel_weight`: