WB microdata coverage gaps (June 2026 audit): EHCVM-2 ×4 + classic-LSMS, tiered by obtainability — post-0.8.0

[ligon]

Summary

A WB Microdata Library coverage audit (June 2026) against our current holdings, classified by whether the microdata is actually obtainable (not just catalogued). Deferred until after the v0.8.0 release — filing now to capture the findings while fresh.

Coverage map at audit time: 47 country config dirs; the audit cross-referenced the live WB catalog (LSMS collection roster + per-study access pages) and confirmed gaps via the library's own data_access.discover_waves() where possible.

Actionability tiers

Tier 1 — WB-direct downloadable (API key suffices) ⭐ highest value, lowest effort

EHCVM wave-2 (2021-22) for 4 WAEMU countries. We already ingest both EHCVM waves for Burkina Faso, Mali, Niger, Senegal — so the harmonization code path exists; these are largely wave-registration + ingest, not new logic. Confirmed local=False by discover_waves('Benin') and downloadable (click-through PUF, no external redirect):

Country	Wave	Catalog	IDNO
Benin	2021-22	6272	BEN_2021_EHCVM-2_v01_M
Côte d'Ivoire	2021-22	6273	CIV_2021_EHCVM-2_v01_M
Guinea-Bissau	2021-22	6274	GNB_2021_EHCVM-2_v01_M
Togo	2021-22	6279	TGO_2021_EHCVM-2_v01_M

Kyrgyz Republic — wholly-missing classic-LSMS country, WB-hosted (login-gated, conditional-use; no external redirect): Multipurpose Poverty Survey 1993 (280), KPMS 1996 (281), KPMS 1998 (283). Larger lift (new country, new harmonization).

Tier 2 — NSO-obtainable (WB redirects to a national agency; obtainable out-of-band)

These are catalog-only on the WB portal (the get-microdata flow redirects to the national statistics office), so add_wave()/the WB API will not fetch them — but they are likely obtainable directly from the NSO, exactly as we already did for GhanaLSS (Ghana Statistical Service) and per CLAUDE.md's "Countries Without Microdata" precedent (Nepal → NSO Nepal):

• Jamaica — Survey of Living Conditions (JSLC), annual ~1988–2000+. WB 2367 (1995) etc. redirect to STATIN/PIOJ (statinja.gov.jm); access needs Jamaican-govt permission. Large annual series if obtained.
• Ecuador — ECV 1995 (2692), ECV 1998 (2693). Redirect to INEC (ecuadorencifras.gob.ec). Note: the JSON data_access block is empty/open-looking, but the external redirect is authoritative — not WB-hosted.

Tier 3 — licensed / fee-gated (escalate; may not be worth it)

• Vietnam — VLSS 1992-93 & 1997-98, VHLSS 2002/2004/2006 (2694 covers the series). Data are property of the Vietnam GSO; permission required, fees $200–$2000, delivered by request. Highest-volume gap (5 waves) but the hardest access gate.

No new LSMS-ISA panel rounds are published yet

For all 8 core LSMS-ISA countries, nothing newer than our latest is in the catalog. Each "next round" is in-field or field-complete but not yet released:

• Ethiopia ESS6, Malawi IHS6, Tanzania NPS6/2026, Uganda 2021-22 & 2023-24 (collected, not yet catalogued — most likely to appear first), EHCVM wave-3 (~2023-24) for all WAEMU (launched Sept 2024).

Action: re-run this audit late 2026 / 2027. data_access.discover_waves(<country>) is the authoritative per-country check.

Verification corrections (catalog claims that are NOT gaps)

Cross-checked against the repo; these looked like gaps but are covered:

• Tanzania NPS4 (2014-15) — covered. tanzania.py lists 2014-15; the 2008-15/ bundle is NPS1–4 (not 1–3).
• Peru — covered. All four ENNIV waves present (1985/1990/1991/1994).
• Serbia and Montenegro 2002/2003 — already scaffolded (countries/Serbia and Montenegro/{2002,2003}/_/); distinct from Serbia/2007.
• Nicaragua — scaffold dir exists (country-level config, no wave data); EMNV waves could be populated but it's not "wholly missing."
• Confirmed not in the WB LSMS collection (don't pursue under this banner): Morocco, Romania, Papua New Guinea, Bolivia, Estonia. South Africa: only ZAF_1993_IHS exists (already covered); no later WB-LSMS waves.

Credentials / prerequisites (for whoever picks this up)

• ✅ WB Microdata API key present (~/.config/lsms_library/config.yml) — the download gate for Tier 1.
• ✅ S3 reader creds present (.dvc/s3_reader_creds.gpg).
• ❌ S3 write/push creds — NOT at the expected .dvc/s3_write_creds + .dvc/config.local (per the add-wave skill), and no AWS_* in env. Needed before dvc push can share new blobs to S3 for others/CI. add_wave(..., push=False) can download + scaffold locally without them.
• Procedure: .claude/skills/add-wave/SKILL.md (+ add-feature for the data_info.yml); discover_waves() / add_wave() / push_to_cache_batch() in lsms_library.data_access.

Suggested sequencing (post-0.8.0)

1. EHCVM-2 ×4 (Benin, Côte d'Ivoire, Guinea-Bissau, Togo) — pilot Benin first to confirm the 2021-22 source structure matches our existing EHCVM ingest, then batch the other 3. Resolve write creds before the push step.
2. Kyrgyz Republic (3 waves) — new-country lift.
3. Tier 2 NSO requests (Jamaica, Ecuador) — out-of-band acquisition like GhanaLSS; manual, async.
4. Vietnam — only if the licensing/fee is judged worth it.

---

Audit method: live WB catalog (LSMS collection roster + per-study get-microdata/JSON access pages) via web research, cross-checked against the repo config tree and data_access.discover_waves(). Re-verify before ingest — the WB catalog updates continuously.

[ligon]

Reconciliation (2026-07-12) — two claims here are now stale, and one is wrong in fact

This audit is good prior art and I should have read it before building the blocked tier (#598) — it independently reached the same obtainability tiering and even cites the Nepal→NSO precedent. Recording the corrections it needs, and what it gives us.

❌ "Kyrgyz Republic — wholly-missing classic-LSMS country"

Not missing. lsms_library/countries/Kyrgyz Republic/ holds 4 waves (1993, 1996, 1997, 1998) and 146 DVC sidecars (~378 MB). The data is downloaded; only the config is absent (no _/data_scheme.yml), which made it invisible to catalog.countries() and therefore to this audit's tooling.

It is one of 10 "unconfigured" countries — data present, zero config — now surfaced by the coverage matrix as a first-class tier (#593). Ethan has ruled them in remit. So Kyrgyz is an add-feature job, not an add-wave one. That's a materially cheaper class of work than this issue assumes.

❌ "South Africa: only ZAF_1993_IHS exists; no later WB-LSMS waves"

True within collection='lsms'. False in fact. South Africa's General Household Survey 2015–2025 (10 waves) sits under repositoryid='datafirst' — a different WB repository, invisible to discover_waves(), which hard-defaults to the LSMS collection.

Same blind spot hides Armenia's 18 waves of Integrated Living Conditions Survey (2001–2018) under repositoryid='central', while data_info.yml listed Armenia as a country "without microdata".

Both now ruled in remit (Ethan, 2026-07-12). Tracked in #597.

This is the sharpest lesson from the reconciliation: this audit's conclusions are correct given its instrument, and its instrument could only see one collection. A negative result from an uncalibrated instrument is a result about the instrument.

✅ What holds, and is valuable

• The obtainability tiering itself — Tier 1 (WB-direct) / Tier 2 (NSO-redirect) / Tier 3 (licensed, fee-gated) — is the right axis, and it is now machinery, not a snapshot: .coder/coverage/blocked_sources.csv (feat(coverage): the blocked tier — wanted, known, unobtainable (+ empty out "Countries Without Microdata") #598), git-tracked, with a mandatory probe URL and blocker, re-probed by bin/coverage_refresh.sbatch, which files an issue when a blocked source comes back.
• Your Tier 2/3 entries belong in it and are not yet there. My initial list has only Nepal (3 waves) + Albania 1996. It should absorb Jamaica (JSLC, STATIN redirect), Ecuador (ECV 1995/1998, INEC redirect), and Vietnam (VLSS/VHLSS ×5, GSO, fees $200–2000). I will add them.
• The EHCVM-2 ×4 (Benin, Côte d'Ivoire, Guinea-Bissau, Togo) Tier-1 finding is independently confirmed by a fresh census — still the highest-value, lowest-effort acquisition on the board.
• "No new LSMS-ISA rounds published yet" — still true; worth re-running late 2026.

Suggested disposition

Keep this open as the acquisition tracker, but re-scope: drop Kyrgyz (it's a config job), fold the Tier-2/3 entries into blocked_sources.csv, and let #597 carry the discovery fix that this audit could not have found.