Phase 0: fix the coverage ruler (blessed tier + absent verdicts + wave-slice grader) by ligon · Pull Request #593 · ligon/LSMS_Library

ligon · 2026-07-12T13:54:48Z

The coverage matrix measures, but it does not measure right. A census (2026-07-11) found that 595 of its 1849 cells — 32% — carry a tier that is wrong or unactionable, and that its errors bias in both directions: it cries wolf about what we have, and stays silent about what we lack.

A workplan built on those numbers is built on sand. This is Phase 0: fix the ruler, then measure.

Design doc: slurm_logs/DESIGN_coverage_discipline_2026-07-11.org

1. `blessed` — adopted (zero library code)

The tier was already fully implemented (loader, promotion, colour, ★ glyph, docs); only its file was a 21-byte header, so the top of the ladder had never once been used.

sane = the automated checks passed. No human has necessarily looked at a single number.
blessed = a human read the actual numbers and believes them.

Schema extended to country,feature,wave,blessed_by,date,note (load_blessed reads via r.get(...), so the provenance columns are free — and they make a blessing auditable rather than a bare assertion).

The rule (CLAUDE.md + docs): if you used a cell in real analysis and looked at its numbers, bless it in the same PR. Seeded empty, deliberately — bulk-blessing the 1208 sane cells would make blessed a synonym for sane and assert a review that never happened.

2. The wave-slice grader applied a COUNTRY-level check — 128 false `builds`

grade_feature grades a wave by slicing the country frame on t, but _check_no_all_null_columns is a country-level check. A question not fielded in a given wave is legitimately all-null within that slice, and was graded builds.

Proof it is a grading bug, not a data bug: of the 124 columns it flagged, zero are all-null country-wide.

is_this_feature_sane gains extra_optional= (default None → historical behaviour byte-for-byte; it has many callers incl. the audit harness). The slice grader passes the set of columns populated somewhere in the parent frame. A column all-null across the whole country still fails — we relaxed the slice, not the country.

Verified on Malawi/Uganda/Niger:


cells compared	434
changed	37 — all `builds → sane`
flips not explained by `no_all_null_columns`	0
regressions	0
`builds` remaining	5 — all `no_null_index_levels` (the genuine `v`-join failures, correctly still red)

The sequencing rule this taught us

A regrade is not a fix. Re-tiering 128 cells to sane would have silently buried two genuine gaps that the buggy grading was catching only by accident. Both were inventoried as issues before this landed:

Nigeria food_prices() silently drops ~99% of rows in 6 of 8 waves (Quantity==0 sentinel → inf → dropped) #591 — Nigeria food_prices() silently drops ~99% of rows in 6 of 8 waves. A Quantity == 0 sentinel (never decoded to NA) makes Price = Expenditure/0 = inf, which transformations.py:933 deletes. 675–1,106 rows survive out of 120k–160k. All six cells were graded sane.
cluster_features: 18 waves across 6 countries have zero GPS (Latitude/Longitude) — will be hidden by the wave-slice grader fix #592 — 18 waves across 6 countries have zero GPS in cluster_features.

Whenever a fix changes how cells are graded rather than what they contain, first inventory what the old grading was catching by accident.

3. `absent` was a black hole — 456 cells

absent said only "not declared for this wave", conflating states that could not be more different. So the number could never reach zero, and every probe's findings evaporated into the prober's head.

Verdicts now live in .coder/coverage/absent_verdicts.csv:

verdict	meaning	closes?
`todo`	data IS there; config missing	no — stays `absent`, but now sized and sourced
`asked-not-distributed`	instrument asked; the shipped extract lacks it	yes → acquisition queue
`not-asked`	genuinely never asked	yes → closed forever
`unsure`	a required check could not run	no — stays in the queue, records why

asked-not-distributed is not hypothetical: Albania 2004's questionnaire lists MODULE 2: DURABLE GOODS, and the variables are simply not in the shipped .dta. Absence in the shipped data is not absence in the instrument — which is why the questionnaire check is mandatory before any permanent close.

Evidence is load-bearing, and this already went wrong

load_verdicts() refuses a closing verdict with an empty evidence field. A closing verdict is a permanent, unsupervised write; an unevidenced negative is unfalsifiable and therefore permanent whether or not it is true.

Albania/_/data_scheme.yml asserted "earlier waves have no shocks module." That claim is false. Verified against source:

wave	source	variable
2005	`Data/migrationE_cl.dta`	`m6e_q00` = "Type of Shock Code" (10 types) + year grid
2008	`Data/Modul_6E_migration_e.sav`	`m6e_q00`, identical
2003	`Data/w2_hh_all.dta`	`B10A_Q10` = "faced any shocks in last 12 months"
2004	`Data/w3_hh_all.dta`	`m10a_q09`, same

Nobody could catch it, because nothing recorded how it was reached. It silently suppressed ~5 cells. Comment corrected; the four waves seeded as todo with citations, 2002 as an honest unsure with its blocker named.

What this does NOT claim

A 38-cell pilot running the full evidence standard returned 25 todo / 13 unsure / 0 not-asked. So absent is not hiding "the survey never asked" — it is hiding a backlog. This mechanism does not shrink the work. It makes the work knowable, sized, and ordered, which is the most an instrument can honestly do.

Tests

12 new: grader exemption, country-wide-null still fails, back-compat default, all four verdicts, and the evidence refusal. All diagnostics/coverage_matrix-touching suites green (243 + 11 + 18 passed).

Still to come in Phase 0

Upstream wave axis (blocked on wave-provenance PR — discover_waves's local flag is currently ~35% wrong)
Surface the 10 in-remit config-less countries as unconfigured
Armenia/Nepal → n/a, not broken (they have no microdata at all)

🤖 Generated with Claude Code

https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe

The `blessed` tier was already fully implemented -- loader, tier promotion, colour, the `★` glyph, docs -- but its file was a 21-byte header, so the top of the ladder had never once been used. `sane` and `blessed` answer different questions, and the gap between them is where quiet errors live: sane the automated checks passed. No human has necessarily looked at a single number. blessed a human read the actual numbers for that cell and believes them. A feature can build cleanly, pass every sanity check, and still be wired to the wrong source column or carry an unverified unit conversion. For anything feeding published analysis, `sane` is not enough. Adopting the tier cost zero library code: - blessed.csv schema extended to `country,feature,wave,blessed_by,date,note`. `load_blessed()` reads via `r.get(...)`, so the provenance columns are free -- and they are what make a blessing auditable rather than a bare assertion. - The rule, documented in CLAUDE.md and docs/guide/coverage.md: if you used a cell in real analysis and looked at its numbers, bless it in the same PR. Seeded empty, deliberately. Blessings accrete; they are never bulk-seeded. An empty blessing file is honest -- bulk-blessing the 1208 `sane` cells would make `blessed` a synonym for `sane`, destroy the only distinction the tier draws, and assert a human review that never happened. Two things found while wiring it: - CLAUDE.md never mentioned the coverage matrix at all (it landed in June and the guide never learned about it). Added a section. - `load_blessed()` calls `pd.read_csv(..., keep_default_na=False)` with no `comment=` argument, so a `#` comment line in blessed.csv parses as a *row* -- a phantom blessed cell. Documented; file kept header-only rather than touching the loader. Also records the corollary that bit two live issues: a `sane` cell is not proof an issue is fixed. The matrix is a cold build, so it structurally cannot see #588's warm-cache-only divergence, and it does not check currency labels (#589). Both look green and are genuinely open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… grader) The coverage matrix measures, but it does not measure right. A census (2026-07-11) found that 595 of its 1849 cells -- 32% -- carry a tier that is wrong or unactionable, and that its errors bias in BOTH directions: it cries wolf about what we have, and stays silent about what we lack. A workplan built on those numbers is built on sand. This is the first half of fixing the instrument. Two defects, and the sequencing between them matters. ## 1. The wave-slice grader applied a COUNTRY-level check (128 cells) `coverage_matrix.grade_feature` grades a wave by slicing the country frame on `t` and calling `is_this_feature_sane` on the slice -- but `_check_no_all_null_columns` is a *country-level* check. A question simply not fielded in a given wave is legitimately all-null within that wave's slice, and was graded `builds` (a defect). It is a grading bug, not a data bug: of the 124 columns it flagged, ZERO are all-null country-wide. Every one is populated in some other wave. Fix: `is_this_feature_sane` gains an `extra_optional=` kwarg (default None -> historical behaviour byte-for-byte; it has many callers). The slice grader passes the set of columns populated somewhere in the parent frame. A column that is all-null across the WHOLE country still (correctly) fails -- we relaxed the slice, not the country. Verified on Malawi/Uganda/Niger: 434 cells, 37 changed, all `builds`-> `sane`, none unexplained by `no_all_null_columns`, ZERO regressions, and the 5 genuinely-broken cells (`no_null_index_levels` -- the `v`-join failure) correctly stayed `builds`. The fix is surgical. ### The sequencing rule this taught us A regrade is NOT a fix. Re-tiering 128 cells to `sane` would have SILENTLY BURIED two genuine gaps that the buggy grading was catching only by accident. Both were inventoried as issues BEFORE this landed: GH #591 Nigeria `food_prices()` silently drops ~99% of rows in 6 of 8 waves. A `Quantity == 0` sentinel (never decoded to NA) makes `Price = Expenditure/0 = inf`, which transformations.py:933 deletes. 675-1106 rows survive out of 120k-160k. All 6 cells were graded `sane`. GH #592 18 waves across 6 countries have ZERO GPS in cluster_features. Whenever a fix changes how cells are GRADED rather than what they CONTAIN, first inventory what the old grading was catching by accident. ## 2. `absent` was a black hole (456 cells) `absent` said only "not declared for this wave", conflating states that could not be more different -- so the number could never reach zero, and every probe's findings evaporated into the prober's head. Verdicts now live in `.coder/coverage/absent_verdicts.csv`: todo data IS there; config missing. Stays `absent` (still work) but is now sized and SOURCED. asked-not-distributed the instrument asked; the shipped extract lacks the variables. An ACQUISITION problem -- a different queue. CLOSES the cell. not-asked genuinely never asked. CLOSES the cell. unsure a required check could not run. Stays in the queue and RECORDS WHY. Silence is never evidence. `asked-not-distributed` is not a hypothetical: Albania 2004's questionnaire lists MODULE 2 DURABLE GOODS, and the variables are simply not in the shipped .dta. Data-side checks cannot see that, which is why C4 (the questionnaire) is mandatory before any permanent close. ### Evidence is load-bearing, and this already went wrong `load_verdicts()` REFUSES a closing verdict with an empty `evidence` field. A closing verdict is a permanent, unsupervised write; an unevidenced negative is unfalsifiable and therefore permanent whether or not it is true. `Albania/_/data_scheme.yml` asserted "earlier waves have no shocks module". It is FALSE. Verified against source: Albania 2005's `migrationE_cl.dta` carries `m6e_q00 = 'Type of Shock Code'` with ten shock types; 2008 is identical; 2003 and 2004 carry "faced any shocks in last 12 months". The claim was wrong, nobody could catch it (nothing recorded HOW it was reached), and it silently suppressed ~5 cells. Comment corrected; the four waves are seeded as `todo` with their file+variable citations, and 2002 as an honest `unsure` with its blocker named. A 38-cell pilot returned 25 todo / 13 unsure / **0 not-asked** -- so `absent` is not hiding "the survey never asked", it is hiding a backlog. This mechanism does not shrink the work. It makes the work knowable. Tests: 12 new (grader exemption, country-wide-null still fails, back-compat default, all four verdicts, and the evidence refusal). Docs in CLAUDE.md + docs/guide/coverage.md. Design: slurm_logs/DESIGN_coverage_discipline_2026-07-11.org Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…the matrix The six defects in the coverage instrument, the four-way adjudication of `absent`, the NOT-ASKED evidence standard (and the pilot that refuted my original premise: 25 todo / 13 unsure / 0 not-asked), and a workplan ordered by leverage. Cited by PR #593. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tier, and stop a scoped run from destroying the snapshot Three more defects in the coverage instrument, plus one found while fixing them. ## Defect 4 — eleven countries were INVISIBLE `catalog.countries()` admits a directory only if it carries `_/data_scheme.yml`, so country dirs holding downloaded microdata and no config never appeared in the matrix at all. That hid **11 countries and 35 waves of already-downloaded data** (Afghanistan, Bosnia-Herzegovina, Brazil, Bulgaria, KenyaLPS, Kyrgyz Republic, Nicaragua, Panama, Peru, Rwanda, Tanzania_Kegera) -- ten of which Ethan has confirmed are in remit. New `unconfigured` tier + `unconfigured_countries()`. They are now visibly red rather than absent from the report. A denominator that omits the work you have not started is not a denominator. ## Defect 6 — `broken` was mis-tiered for the known-no-microdata countries All 8 `broken` cells in the cube were Armenia (2) and Nepal (6) -- both already documented in CLAUDE.md as having no source data in the repo. They cannot build; that is a data-availability fact, not a defect. Declared in the canonical `data_info.yml` (config, not code -- GH #436) and graded `n/a`. This matters beyond tidiness: the most alarming tier in the ladder contained *nothing actionable*, so a genuine build failure would have been lost among them. ## Found while fixing the above: a scoped run DESTROYED the snapshot `save_snapshot()` wrote wholesale, so `make matrix C="Uganda"` -- which grades only Uganda -- silently erased every cell it did not grade. `docs/guide/coverage.md` documents exactly that as the "spot refresh" procedure, immediately followed by `git add .coder/coverage/latest.csv && git commit`. Following the guide verbatim therefore committed a 67-cell snapshot over the authoritative 1849-cell one. I did it to myself while testing the renderer. `save_snapshot(..., merge=True)` (now the default) upserts on `(country, feature, wave)` and leaves un-graded cells untouched. A full sweep still rewrites everything it grades, so the authoritative run is unaffected -- it simply no longer depends on *not* being scoped. **A partial measurement must never be able to erase a complete one.** ## Notes - Two full-suite failures investigated, neither caused by this work: `test_feature_ghana_per_wave` is pre-existing and already tracked as **GH #589** (its title is literally that test name); the Albania `sample` failure was a parquet-cache race from running three heavy builds concurrently right after a `data_scheme.yml` edit invalidated Albania's content hash. Passes in isolation. - 11 new tests (34 in test_coverage_matrix.py total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s wrong Addendum #3 claimed "`PYTHONPATH` alone does NOT redirect imports of `lsms_library` to a worktree". That is FALSE -- PYTHONPATH does beat the `.pth`. The wrong claim sent agents to a needless mitigation (build a fresh venv inside the worktree, or verify by static diff only). The ACTUAL trap is `python <script>.py`. Python sets `sys.path[0]` to the SCRIPT's own directory, not cwd -- so `cd`-ing into the worktree does not protect a script run: cwd never enters `sys.path` at all, the `.pth`'s main-repo path wins, and you import the MAIN checkout while believing you are testing the worktree. Verified (cwd = worktree throughout): python -c "import lsms_library" PYTHONPATH unset -> worktree (cwd is sys.path[0]) python bench/scan.py PYTHONPATH unset -> MAIN CHECKOUT <-- the trap python bench/scan.py PYTHONPATH=<wt> -> worktree This is nastier than the old text described, because the agent has done everything that LOOKS right (cd'd into the worktree, run the worktree's own script) and still tested the wrong code -- silently. It has already cost a worker agent a full wasted audit run (found while reviewing PR #594). Since the failure is silent and looks like success, the guidance now says to ASSERT it rather than assume it: import lsms_library assert 'worktrees' in lsms_library.__file__, lsms_library.__file__ Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…" claim was FALSE I wrote, in the design doc, in commit 8dcd2bc's message, and in PR #593: "none of the 136 wave directories records the WB catalog id it came from ... not one pre-existing wave has one (find ... -name SOURCE.org -> 0)" That is false. **117 of the 136 waves had a SOURCE.org all along.** My `find` used `-maxdepth 3`; the file lives at `{Country}/{wave}/Documentation/SOURCE.org` -- depth **4**. The command returned 0 because it could not see the files, not because they were absent. I ran one command, it returned zero, and I called it proof. That is exactly the *unevidenced negative* this very design doc spends its evidence-standard section warning against -- the same failure as Albania's "earlier waves have no shocks module". The rule applies to me too: silence is not evidence, and a null result from an instrument you have not validated is a null result *about the instrument*. The real defect is WORSE than the one I described. The provenance is not merely missing -- in places it is WRONG: Nigeria/2018-19/Documentation/SOURCE.org -> catalog 3827 (Nigeria LSS) Nigeria/2018-19/Data/ -> aux_plantingw4, aux_harvestw4 = GHS-Panel W4 (3557) A backfill that TRUSTED SOURCE.org would have cemented that error rather than fixed it. Absent provenance is a gap; wrong provenance is a lie, and it is the harder failure. PR #595 therefore validates rather than trusts, and keeps `none` (definitively not a WB dataset) distinct from `unknown` (we don't know) -- collapsing those would recreate the `absent`-black-hole mistake of Defect 1 one level down. Also records a trap found while implementing it: **SOURCE.org is load-bearing.** `Country.waves` (country.py:1452) falls back to scanning for dirs containing `Documentation/SOURCE.org`, so *creating* one PROMOTES a directory into a wave. Stamping all 136 broke `test_sample.py::test_covers_all_waves[Benin]`, because `Benin/2018-2019/` is a stray duplicate of `Benin/2018-19/` and stamping it promoted the duplicate into a wave `sample()` does not cover. Metadata that silently changes the shape of the data model is not metadata. Credit: caught by the wave-provenance agent (PR #595), not by me. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ligon · 2026-07-12T15:03:40Z

Correction: my "no wave records its provenance" claim was FALSE

This PR's description, and commit 8dcd2bcb's message, assert:

"none of the 136 wave directories records the WB catalog id it came from ... not one pre-existing wave has one (find ... -name SOURCE.org → 0)"

That is wrong. 117 of the 136 waves had a SOURCE.org all along.

My find used -maxdepth 3. The file lives at {Country}/{wave}/Documentation/SOURCE.org — depth 4. The command returned 0 because it could not see the files, not because they were absent.

I ran one command, it returned zero, and I called it proof. That is precisely the unevidenced negative this PR spends its evidence-standard on — the same failure as Albania's "earlier waves have no shocks module." The rule applies to me: a null result from an instrument you have not validated is a null result about the instrument.

Corrected in e1a0d17 (design doc). Caught by the wave-provenance agent (#595), not by me.

The real defect is worse than the one I described

The provenance is not merely missing — in places it is wrong:

Nigeria/2018-19/Documentation/SOURCE.org  ->  catalog 3827  (Nigeria Living Standards Survey)
Nigeria/2018-19/Data/                     ->  aux_plantingw4, aux_harvestw4
                                              = GHS-Panel W4 (3557)  -- a DIFFERENT survey

A backfill that trusted SOURCE.org would have cemented that error rather than fixed it. Absent provenance is a gap; wrong provenance is a lie, and it is the harder failure. #595 validates rather than trusts.

And a trap worth knowing: `SOURCE.org` is load-bearing

Country.waves (country.py:1452) falls back to scanning for directories containing Documentation/SOURCE.org — so writing one PROMOTES a directory into a wave. Stamping all 136 broke test_sample.py::test_covers_all_waves[Benin], because Benin/2018-2019/ is a stray duplicate of Benin/2018-19/ (identical .dta) and stamping it promoted the duplicate into a wave sample() doesn't cover.

Metadata that silently changes the shape of the data model is not metadata.

Nothing in this PR's code depends on the false claim — the grader fix, the absent-verdicts mechanism, unconfigured, and the no-microdata tier are all unaffected. The claim appeared only in the motivating prose for Defect 3, which is implemented in #595, not here.

`Benin/2018-2019/` was a `Data/`-only staging duplicate of the configured `Benin/2018-19/` wave. Removed at Ethan's direction (2026-07-12). Verified safe before deleting: - all **52** DVC sidecars are **md5-identical** to `2018-19/Data/`'s; - **zero** files unique to either directory; - no `_/` config, no `Documentation/`, not in `Country('Benin').waves`; - referenced by nothing but the `Benin/_/CONTENTS.org` note describing it. The DVC blobs are content-addressed and shared, so removing the duplicate sidecars orphans no data. ## It was not merely clutter — it was a trapdoor `Country.waves` (`country.py:1452`) falls back to scanning for directories containing `Documentation/SOURCE.org`, so **writing a SOURCE.org promotes a directory into a wave**. A provenance-backfill pass that stamped every directory therefore turned this duplicate into a wave that `sample()` does not cover, breaking `test_sample.py::test_covers_all_waves[Benin]` (found in PR #595). Removing the directory closes that trapdoor rather than working around it. Post-delete: `Country('Benin').waves == ['2018-19']`, `sample()` (8012, 5), `pytest tests/test_sample.py -k Benin` 9 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ources Backlog reconciliation, and a prior-art miss I should own: **GH #552 (June 2026) already audited WB coverage gaps "tiered by obtainability"** — Tier 1 (WB-direct) / Tier 2 (NSO-redirect) / Tier 3 (licensed, fee-gated) — and even cites the Nepal->NSO precedent. I built the `blocked` tier without reading it. CLAUDE.md tells me to run the prior-art-ledger skill at the start of a non-trivial task; I skipped it. The reconciliation was productive in BOTH directions. ## What #552 gives us: `blocker_kind` #552's tiering is the right axis, and it belongs in the machinery as STRUCTURE rather than prose, because the kinds route to completely different people: nso-redirect the WB catalogues it but redirects to a national statistics office whose channel is broken. -> chase a human. fee-gated obtainable, for money and permission (Vietnam GSO: USD 200-2000 + consent). -> a BUDGET decision, not an engineering one. NEVER queue this for an RA. unconfirmed we believe it is blocked but have not proven it; it may be a plain obtainable TODO (Albania 1996). The honest state when you have not actually checked. "The site is down" and "it costs $2,000" are not the same blocker. Flattening them is how a money question silently becomes an engineering backlog item that nobody can ever close. blocked_sources.csv grows from 4 rows to 8, absorbing #552's Tier 2/3: Jamaica (JSLC, STATIN redirect), Ecuador (ECV 1995/1998, INEC redirect), and Vietnam (VLSS/VHLSS x5, GSO, fee-gated). They are now git-tracked, probed, and re-checked — not a snapshot in an issue that ages quietly. ## What we give #552: two of its claims are stale, one is wrong in fact - **"Kyrgyz Republic — wholly-missing"**: NOT missing. 4 waves, 146 DVC sidecars (~378 MB) are already downloaded; only the CONFIG is absent. It is one of the 10 `unconfigured` countries (#593). An add-feature job, not an add-wave one — a materially cheaper class of work than #552 assumes. - **"South Africa: no later WB-LSMS waves"**: true within `collection='lsms'`, FALSE in fact. The General Household Survey 2015-2025 sits under `repositoryid='datafirst'`. Same blind spot hides Armenia's 18 ILCS waves under `repositoryid='central'`. Both now in remit; see GH #597. #552's conclusions are correct GIVEN ITS INSTRUMENT, and its instrument could only see one collection. *A negative result from an uncalibrated instrument is a result about the instrument.* That is the third time that exact lesson has landed in this series -- twice on me. Also closed as verified-fixed during the sweep: #571 (Guatemala), #572 (Serbia), #573 (Panama — its data_scheme.yml landed today in a0ba3e1), #574 (GhanaSPS food_acquired; its sibling #579 stays open — GhanaSPS.sample() still raises). And corrected #587, whose stated hypothesis for its largest bloc (Nigeria's 22 `dropped` cells as "false positives") is refuted: the built `t` values are valid wave labels from the OPPOSITE PP/PH round. They are real half-missing features, not a reporting artefact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ually lives `discover_waves()` searched a single hard-coded collection (`lsms`). Whole household-survey series that the World Bank publishes under OTHER repositories were therefore not "not yet fetched" but structurally unfindable: * Armenia's Integrated Living Conditions Survey -- 18 annual waves, 2001-2018 -- sits in `central`. `lsms` carries only the 1996 Household Budget Survey. Armenia was recorded as a country "without microdata". * South Africa's General Household Survey -- 21 waves, 2002-2025 -- sits in `datafirst`. `lsms` carries only the 1993 Integrated Household Survey. That is 39 waves of in-remit data invisible behind a default argument, and it made the coverage matrix's "honest denominator" (#593, #595) confidently wrong: it reported no missing waves for Armenia while 18 sat in the catalog. The fix is a per-country `repositories` list on CountryCatalog, defaulting to ("lsms",), unioned and deduplicated on the catalog id. 34 of 36 countries are unaffected. Deliberately NOT a removal of the collection filter. Searching everything inflates results 30-400x (Findex, DHS, Afrobarometer, enterprise surveys; `datafirst` alone returns 320 South African rows -- censuses, election studies, school registers, media surveys). That trades a false-negative problem for a false-positive one, and the second is worse: it buries the real gaps in noise. So widening is paired with an `idno_pattern` that pins the survey series -- the lever #595 already built. The two compose: `repositories` says where to look, `idno_pattern` says what counts. Both are config, not heuristics. The pin also defends against a trap that id-matching alone cannot see: the WB catalogues some studies TWICE, in two repositories, under two different ids. `central` 3016 (MWI_2010_IHS-III_v01_M_v01_A_ML) is the same Malawi IHS3 as `lsms` 1003, which we hold as Malawi/2010-11/; `datafirst` 902 (ZAF_1993_PSLSD) is the same 1993 survey as `lsms` 297 (ZAF_1993_IHS), which we hold as South Africa/1993/. Nothing in the catalog metadata links those pairs. Naive widening reports both as missing waves. They are not. Armenia stays listed under "Countries Without Microdata" -- we still hold zero data files, and delisting it would flip its coverage cells from `n/a` to `broken`. What changes is the reason: an acquisition backlog, not an absence. No data is acquired here; that is a separate `add-wave` job. `Country.waves` is byte-identical to pristine for all 36 countries / 108 wave dirs. Refs #597. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe

ligon and others added 6 commits July 12, 2026 06:27

ligon merged commit f7b5291 into development Jul 12, 2026
10 checks passed

ligon deleted the feat/coverage-blessed branch July 12, 2026 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase 0: fix the coverage ruler (blessed tier + absent verdicts + wave-slice grader)#593

Phase 0: fix the coverage ruler (blessed tier + absent verdicts + wave-slice grader)#593
ligon merged 7 commits into
developmentfrom
feat/coverage-blessed

ligon commented Jul 12, 2026

Uh oh!

ligon commented Jul 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ligon commented Jul 12, 2026

1. blessed — adopted (zero library code)

2. The wave-slice grader applied a COUNTRY-level check — 128 false builds

The sequencing rule this taught us

3. absent was a black hole — 456 cells

Evidence is load-bearing, and this already went wrong

What this does NOT claim

Tests

Still to come in Phase 0

Uh oh!

ligon commented Jul 12, 2026

Correction: my "no wave records its provenance" claim was FALSE

The real defect is worse than the one I described

And a trap worth knowing: SOURCE.org is load-bearing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

1. `blessed` — adopted (zero library code)

2. The wave-slice grader applied a COUNTRY-level check — 128 false `builds`

3. `absent` was a black hole — 456 cells

And a trap worth knowing: `SOURCE.org` is load-bearing