Tanzania: duplicate index entries in cluster_features and shocks

[ligon]

Summary

Tanzania's cluster_features and shocks tables have massive duplicate index entries.

cluster_features — 80% duplicates

data_scheme.yml declares index: (t, v) (one row per cluster-wave), but the aggregate var/cluster_features.parquet contains household-level granularity that hasn't been deduplicated to cluster-level. Result: 2656/3320 (80%) duplicate (t, v) entries.

shocks — 60% duplicates

data_scheme.yml declares index: (t, i, Shock), which should be unique (one row per household × wave × shock type). The actual parquet has 246,479/407,486 (60.5%) duplicate entries. Likely cause: the 2008-15 multi-round data contains panel household IDs that produce duplicates after concatenation across rounds.

Affected tests

• test_no_duplicate_rows[Tanzania/cluster_features]
• test_no_duplicate_rows[Tanzania/shocks]
• test_feature_is_sane[Tanzania/cluster_features]
• test_feature_is_sane[Tanzania/shocks]

Fix

• cluster_features: The wave-level scripts should aggregate to (t, v) granularity before writing, or the country-level aggregation should deduplicate.
• shocks: The 2008-15 wave script needs to assign distinct t values per round to avoid duplicate (t, i, Shock) entries, similar to how other multi-round scripts handle this (see CLAUDE.md § PP/PH Countries).

🤖 Generated with Claude Code

[ligon]

Verified as of commit 682825d: Tanzania cluster_features shape=(2389, 3) duplicate_index_entries=0; shocks shape=(162928, 5) duplicate_index_entries=0. Both v-migration and other recent work have resolved the duplicate index issue.