feat(provenance): resolve the last 5 unknown waves from evidence (0 unknown)

[ligon]

Follow-up to #595 (merged). Closes out the 5 waves that PR left as unknown.

Note: #595 was merged and its branch deleted while I was preparing this, so this is a new PR on a new branch rather than a push to #595.

The 5 unknowns are now 0

You did the search; I re-validated every candidate independently — against the directory's actual contents and the live WB API — before applying. A year match alone proves nothing; that is exactly what put the wrong id in Nigeria/2018-19. All five checked out.

Resolved to a catalog id (3)

wave	id	evidence
Niger/2011-12	2050	Data/ contains a directory literally named NER_2011_ECVMA_v01_M_Stata8 — the WB idno itself. Confirmed id=2050, idno=NER_2011_ECVMA_v01_M, 2011-2012.
Senegal/2021-22	6278	58 ehcvm_*_sen2021 files (ehcvm_conso_sen2021, ehcvm_individu_sen2021, …). Confirmed SEN_2021_EHCVM-2_v01_M, the only SEN EHCVM-2 entry (4297 is 2018-19).
Nepal/2003-04	74	NPL_2003_LSS-II_v01_M, idno + years unambiguous — but see below.

Reclassified unknown → none (2)

GhanaSPS/2013-14 and GhanaSPS/2017-18 — EGC-ISSER (Yale/ISSER) panel waves 2 and 3.

I re-ran the absence check independently rather than take it on faith, since "absent" is precisely the claim that goes wrong quietly:

• A GHA query across the entire WB catalog (no collection filter, 152 entries) returns exactly one GSPS study — 2534, the 2009-2010 wave — and nothing later.
• A global EGC-ISSER keyword sweep surfaces 8 hits, none of them a later Ghana panel wave.
• File naming (01a_consent.dta, 01b2_roster.dta, 00_comm_info.dta) is the EGC-ISSER instrument.

none is the right terminal state. Agreed on the reasoning: leaving them unknown means someone re-searches them forever — the same absent-vs-unknown black hole this work exists to close, one level down.

Nepal: provenance known, data absent — kept as two facts

You flagged this and you were right to. Nepal/2003-04/Data/ holds zero files, so the id is catalog-matched but not content-validated. Rather than bury that in prose, it gets a structured field:

#+PROVENANCE_VALIDATION: — content-validated (the wave's own data files confirm the id: Nigeria, Niger, Senegal) vs. catalog-only (catalog metadata alone; no local data to check: Nepal).

"Right for the right reason" and "right by coincidence" must not look identical in the record — that conflation is the whole bug this series is about.

#+CATALOG_ID: 74
#+CATALOG_IDNO: NPL_2003_LSS-II_v01_M
#+PROVENANCE_SOURCE: worldbank
#+PROVENANCE_VALIDATION: catalog-only
#+PROVENANCE_NOTE: ... NOT content-validated: Nepal/2003-04/Data/ is EMPTY -- Nepal's
  microdata is hosted by the Nepal NSO rather than the World Bank ... So the provenance
  is known while the data is absent; these are two different facts and are recorded as such.

A bug the new test caught

Writing test_no_shipped_source_org_is_left_unknown surfaced a real defect in my own _may_write guard: Country.waves is not always a list of directory names. Nigeria's are survey rounds (2010Q3, 2011Q1) and Tanzania's are logical waves (2008-09…2014-15); both map onto directories via wave_folder_map. So 14 perfectly real wave folders looked "undeclared". _declared_waves() now folds wave_folder_map values in. No output change (all those dirs already had a SOURCE.org, so they took the safe update path), but it was a live footgun for the next wave added.

Final state

wave dirs scanned : 135
stamped           : 123
    resolved      : 123
        source-url    : 52
        doi-lookup    : 48
        external      : 17
        override      :  6
    unknown       :   0
skipped           :  12   (not declared waves; stamping would create one)

Verification

• Country.waves identical to pristine for all 47 countries (the feat(provenance): record each wave's WB catalog id; match discover_waves on it #595 invariant — not regressed).
• Backfill remains idempotent, proven byte-for-byte across consecutive runs of all 123 files.
• Full suite: 3638 passed, 0 failed (test_currency.py::test_feature_ghana_per_wave deselected — it fails identically on development; pre-existing, unrelated).
• New tests: test_no_shipped_source_org_is_left_unknown (asserts all 123 record a terminal answer), test_evidence_resolved_waves (pins the 4 id resolutions + their validation levels), test_ghana_sps_egc_isser_waves_are_none_not_unknown (pins none, not unknown). 30 tests in test_wave_provenance.py; 71 with test_data_access.py.
• Rebased on development, which includes the Benin/2018-2019/ deletion (facdc7b6) — the stray duplicate feat(provenance): record each wave's WB catalog id; match discover_waves on it #595 surfaced.

Remaining

The 12 skipped dirs are not declared waves: Bulgaria ×5, Afghanistan ×2, Brazil/1996-97, KenyaLPS ×4. They hold data but the library does not expose them as waves, so they are deliberately left unstamped (stamping would create waves — the SOURCE.org-is-load-bearing trap from #595). Worth a separate look: either wire them up as real waves, or delete them as Benin/2018-2019 was.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe