fix(Tanzania): describe a cluster from inside it; re-key 2020-21 (GH #323, Site 2)

[ligon]

Fixes the Tanzania cluster_features cluster key — the worst Site-2 cell in the
corpus (2,104 of 7,167 cluster-attribute cells contested, 29.4%; 65.4% in
2012-13). Config only: nothing under lsms_library/*.py, no aggregation:
keys (D1).

The cause is not an ambiguous cluster code

Every wave read Region / District / Rural off the household cover
page — the address where the household was interviewed. The NPS is a
tracking panel: it follows movers and split-offs, records their new
address, and carries their original sampling cluster v forward unchanged. So
the geo columns are HOUSEHOLD attributes, v is a CLUSTER attribute, and the
projection onto (t, v) asks a household that has left the cluster to describe
it.

Round 1 settles that these really are cluster attributes: in 2008-09, before
anyone has moved, all 409 clusters agree on all three columns — 0 contested.
Disagreement appears only from round 2 and tracks movement (549 / 1,154 / 314
households in rounds 2/3/4 report a region that is not their cluster's).

The identifier already carries the answer

The NPS cluster id is a hierarchical geocode whose leading fields are the
cluster's region and district. Validated on the 2008-09 frame:

• all 26 region codes present map to exactly one region name (0 violations);
• the geocode's district digit equals the reported district code ha_02_1 for
  100.0% of rows.

Field widths are a property of the wave, never of the string's length —
2019-20's 8-character '11014002' is 01-1-014-002 = DODOMA under its own
9-digit scheme and IRINGA under the 8-digit one. cluster_region() therefore
makes the caller name the scheme, and a test pins both readings.

The fix

Each wave excludes the households that no longer live in the cluster, in tiers
against the cluster's own geocode (region+district → region → unfiltered, so
tightening the test can never drop a cluster). Tiers are evaluated per (t, v),
not per v — grouping on v alone silently lost 12 cluster-waves.

The frame stays at household grain on purpose: the projection belongs to
Wave.cluster_features where the framework audits it, and collapsing in config
would drive the #323 instrument to 0 for free. A test pins that i survives
into the projection.

Two waves needed more than a filter:

• 2019-20 was reading the mover question. hh_a01_1 / hh_a02_1 are asked
  only of households found at a NEW location — null for 89.9% of those found
  at their ORIGINAL location, i.e. exactly the ones that can describe the
  cluster. Re-sourced to t0_region / t0_district. Separately the 326
  SPLIT-OFF rows (sdd_cluster blank) carry a clusterid they were never
  sampled in — under one DODOMA code they pool households whose own strata span
  DAR ES SALAAM, DODOMA and ZANZIBAR — which manufactured 100 clusters that do
  not exist (247 → 147).
• 2020-21's key was simply wrong. cluster_features keyed on clusterid
  while sample has always keyed on y5_cluster, so all 417 of its (t, v)
  pairs were unknown to sample — _join_v_from_sample matched nothing.
  clusterid is also blank for all 545 booster households, deleted outright
  on a NaN key. v → y5_cluster: 418 → 515 clusters, 0 unknown to sample.

Numbers

Cold build, isolated LSMS_DATA_DIR with dvc-cache symlinked to the shared L1
(LSMS_NO_CACHE=1 alone is not enough — it is soft for the script-path
2008-15 L2-wave parquet).

wave	contested before	after	clusters
2008-09	0	0	409 → 409
2010-11	329	78	409 → 409
2012-13	803	147	409 → 409
2014-15	160	41	498 → 498
2019-20	257	72	247 → 147
2020-21	557	47	418 → 515
total	2,106	385	2,389 → 2,387 rows

By column: Region 706 → 3, District 758 → 118, Rural 642 → 264. Rows
destroyed by the collapse (the framework's own Site-2 audit) 10,982 → 2,687;
NaN-key deletions 545 → 0.

sample.v ≡ cluster_features.v: pairs unknown to sample are 0 in every
wave except 2014-15's 5, which are pre-existing — present in the
wave-level sample.parquet, lost in _finalize_result's id_walk.

Rejected, on evidence

Full write-up in .coder/ledger/323-tanzania-key.md.

• A composite key (Uganda's fix) — zeroes Region tautologically by
  splitting each real cluster across the regions its movers went to: 2012-13 goes
  409 clusters → 1,165, and v desynchronises from sample.
• strataid2 as the cluster-invariant source — shows zero within-cluster
  variation in all four rounds because it is 100% NULL in rounds 1-3. The
  invariance was an artefact of missingness.
• Carrying attributes from each cluster's first-appearance round — the 82
  clusters first appearing in 2012-13 are pre-2012 clusters renumbered after
  the Simiyu/Geita/Njombe/Katavi region splits, so it is not a frame round.
• Decoding urban/rural from the geocode — predicts urb_rur for only 75.3%
  of round-1 clusters.

Residue, not forced to zero

• District (118) — overwhelmingly two names for one place: MBINGA (NYASA) /
  MBINGA(NYASA) / MBINGA NYASA / NYASA, typos MUFINDI/MFINDI, and
  mid-panel district splits (Chalinze/Bagamoyo, Kibiti/Rufiji, Ubungo/Kinondoni).
  A harmonize_district table would close most of it.
• Rural (264) — EAs whose households are genuinely not all urban or all
  rural; modal share 0.75–0.88.
• Region (3) — post-2012 region boundaries.
• 13 of 2,387 clusters end up with a Region that is not the one their geocode
  encodes: 4 are Simiyu clusters still wearing a pre-2012 Shinyanga code (the
  households are right, the id is stale) and 3 are y5_cluster ids malformed
  in the source, carried through rather than repaired by guesswork.

Deferred

• 2019-20 sample.v for the 326 split-offs now dangles. Their clusterid is
  not a sampling cluster; blanking it changes v on every 2019-20 household
  table for 27.5% of the wave — a semantic change that belongs in its own PR.
• harmonize_district; a cluster-level Rural classification.

Tests

tests/test_gh323_tanzania_cluster_key.py — 28 tests, config-level only.

Negative control: against a pristine origin/development config tree
(git archive into a separate LSMS_COUNTRIES_ROOT + data dir), 20 of 28
fail — every contested-cell ceiling, the sample.v ≡ cluster_features.v
test, both 2020-21 key tests, and the per-wave cluster counts.

test_tanzania_grain_gh323.py (19), test_tanzania_community_cluster_xwalk.py,
test_schema_consistency.py and test_gh323_benin_togo.py all pass.

Pre-existing and untouched:
test_gh323_explicit_reducers.py::test_core_does_not_dispatch_the_reducers
fails on pristine development too (core's private
_collapse_to_cluster_grain contains a banned substring) — also recorded in
.coder/ledger/323-uganda.md §8.

🤖 Generated with Claude Code

[ligon]

Adversarial review — PR #642 (Tanzania cluster_features, GH #323 Site 2)

Verdict: APPROVE_WITH_NOTES. Every headline number reproduced cold and independently; two attacks designed to break the central claim both came back supporting it. The notes below are claim framing, one small minority-rules edge case, and a cosmetic label regression.

Method: fresh worktree at 38be590e, isolated LSMS_DATA_DIR (dvc-cache symlinked to the shared L1), LSMS_COUNTRIES_ROOT pointed at the worktree config tree — L2 rebuilt from scratch for both the PR tree and a pristine origin/development (45aee170) tree in a second isolated data dir. No dvc CLI invoked anywhere.

Reproduced cold, from scratch

	measured on development	measured on this branch	PR claims
contested cells, by wave	0 / 329 / 803 / 160 / 257 / 557 = 2,106	0 / 78 / 147 / 41 / 72 / 47 = 385	identical
by column	Region 706, District 758, Rural 642	Region 3, District 118, Rural 264	identical
clusters	409/409/409/498/247/417	409/409/409/498/147/515	identical (body says 418 for dev's 2020-21; it is 417)
(t,v) unknown to sample	2020-21 417 of 417, 2014-15 5	0 everywhere, 2014-15 5	identical

Independently re-derived the load-bearing source facts rather than trusting them:

• round-1 geocode ↔ frame: region agreement 1.000, district-digit ↔ ha_02_1 agreement 1.000, geocode-NA rate 0.000, 26 region codes — the crosswalk claim is exact;
• 2019-20 hh_a01_1 null for 89.99 % of HOUSEHOLD FOUND - ORIGINAL LOCATION rows (17.3 % among NEW-location rows) — the "it is the mover question" diagnosis is right; 326 split-off rows; 100 clusterids occur only among split-offs (so 247 → 147 is exactly the removal of manufactured clusters, not a loss of real ones);
• 2020-21 clusterid isna for exactly 545 rows, all of them BOOSTER SAMPLE, 417 distinct; y5_cluster 515 distinct = 447 non-booster + 68 booster, zero overlap. Both claims hold.
• Negative control verified: the new test file run against a pristine development config tree gives 20 failed, 8 passed, exactly as stated. On the PR tree: 28/28 pass cold. Also green: test_tanzania_grain_gh323, test_tanzania_community_cluster_xwalk, test_grain_collapse*, test_gh323_grain_contract (60), and test_schema_consistency + test_table_structure (298).

What I tried in order to break it, and could not

1. Null control: is the improvement just "fewer rows"? The filter deletes 8,295 rows overall, and a smaller group is mechanically less likely to disagree. So I re-ran the #323 instrument after dropping, per cluster, the same number of rows at random (5 replicates):

wave	before	random-drop control	actual after
2010-11	329	282–298	78
2012-13	803	741–763	147
2014-15	156	142–148	39

Random deletion buys almost nothing. The filter is selecting a meaningful subpopulation, not thinning one. This is the strongest evidence in favour of the PR and it is not in the PR — worth adding.

2. Is "movers" a story or a fact? Tested independently of the geocode, on the panel key UPHI (note: r_hhid is not unique within a round — 12,710 dupes — so it cannot be used for this):

• P(region-mismatch in 2012-13 | mismatch in 2010-11) = 0.876, base rate 0.100
• P(region-mismatch in 2014-15 | mismatch in 2012-13) = 0.960, base rate 0.192

Mismatch is a near-absorbing household state, which is what tracked out-migration looks like and is not what per-round coding noise looks like. Diagnosis corroborated.

3. Invariance by missingness? No — geocode-NA rate is 0 in all four 2008-15 rounds; clusterid string lengths are 7/8/11/12 only, all handled by the declared schemes. The strataid2 trap the ledger records is real and correctly avoided.

4. A fabricated finer key? No. v is unchanged in 2008-15 and 2019-20; the single re-key (2020-21 clusterid → y5_cluster) reduces disagreement with sample from 417/417 to 0 rather than inflating cluster counts. Scope is config-only: no lsms_library/*.py, no aggregation: key, no reducer, and i demonstrably survives into the projection (the GrainCollapseWarning still fires on the residue).

Notes

N1 (medium — claim framing). "Region 706 → 3" is true but circular. The filter keeps exactly the rows whose reported Region equals cluster_region(v), so in any cluster with ≥ 1 matching row, Region-contested is 0 by construction; the surviving 3 are just the tier-3 clusters. That number measures the filter's definition, not the fix's quality — precisely the "duplicate counts falling to zero is not evidence" pattern. The claim survives only because the non-circular evidence exists (round-1 validation at 1.000, and the null control above). Please say in CONTENTS.org that Region → 0 is by construction and lead with District/Rural + the null control.

N2 (medium — behaviour). The tier rule is "any matching row wins", with no plurality guard, and it demonstrably mislabels two clusters. keep = ok_fine | (~has_fine & ok_region) | … means a single geocode-matching row outvotes any number of unanimously dissenting residents. Verified end-to-end in the built table:

• 2014-15 / 19604003: 22 rows unanimously report GEITA, 2 report Mwanza/Sengerema → kept 2 → finished Region = Mwanza.
• 2014-15 / 19604004: 15 report GEITA, 3 report Mwanza/Sengerema → kept 3 → finished Region = Mwanza. The same cluster's District in 2012-13 is literally Geita — i.e. geocode district 19-6 was Geita District, which became Geita Region in 2012, and the Mwanza-matching rows match only because Mwanza's district codes were renumbered after Geita left.

Full sweep of 2008-15: 8 clusters where a minority outvotes ≥ 4 unanimous dissenters (2010-11 ×2, 2012-13 ×4, 2014-15 ×2). Small (8 of 2,387), but CONTENTS.org's "the config lets the residents win" is only true at 100 % dissent (tier 3, e.g. 17201000 / 17201001 → Simiyu, correct). Either add a plurality guard to tier 1/2 or state the rule accurately.

N3 (low — regression). The 2020-21 hook stores an uppercased place name and thereby leaks 5 new ALL-CAPS District labels — BUSOKELO, CHALINZE, KONDOA URBAN, MBALALI, NZEGA TOWN (8 rows) — where development renders them title-case; they are labels the downstream mapping does not cover. tanzania.normalize_place()'s own docstring says the normalized form is "for COMPARISON only — never stored". No cross-wave collision (none of the five occurs in another wave), so it is cosmetic, but a PR about label consistency should not mint new variants. Fix: normalise for the comparison, keep the source casing in the frame.

N4 (low — disclosed). 2019-20 now leaves 281 of 1,184 sample households (23.7 %) with a v that has no cluster_features row. Correctly deferred in the PR, and it replaces a much larger dangling set — on development all 4,709 2020-21 households dangle — so the net is strongly positive. Please open the follow-up issue so it does not evaporate.

N5 (low — doc). Internal number drift: the 2008-15 script comment says contested go to "0 / 78 / 144 / 35" while the ledger, CONTENTS.org, the tests and my measurement all say 0 / 78 / 147 / 41; the body says dev's 2020-21 has 418 clusters (it is 417); the abstract says 2,104 vs the measured 2,106. Also "no cluster lost" is true, but 3 2014-15 cluster-waves reach tier 3 and are left unfiltered — worth saying.

N6 (low — hygiene). The two mapping.py files are the only sys.path.insert(0, …) in the whole countries tree, and they put Tanzania/_/ (which contains generically-named modules — shocks.py, livestock.py, panel_ids.py) ahead of everything for the rest of the process. No collision exists today (checked: no country config imports those bare names), but append is the safer idiom here.

None of N1–N6 blocks the merge.