feat(coverage): the blocked tier — wanted, known, unobtainable (+ empty out "Countries Without Microdata")#598
Merged
Merged
Conversation
The state between "obtainable" and "does not exist": the data EXISTS, we KNOW
where it is, and we CANNOT get it because the acquisition channel is broken.
Nepal (3 waves) NSO Nepal hosts the microdata, not the WB. Their site is
unreachable and no human contact has been reached.
Albania 1996 directory + WB catalog entry (2311, ALB_1996_EWS) both exist;
zero data files.
## I had this wrong, and the mistake was instructive
Two commits ago I graded these `n/a` under "Countries Without Microdata"
(Defect 6). But `n/a` means "no readiness applies" -- it reads as *nothing to
see here*, and it BURIED a gap we actively want. That is the exact lumping
error this module exists to prevent, committed by its author, in the fix for a
different lumping error.
`blocked` is *visible* (it counts against us) but *not actionable by us* --
nobody closes it by trying harder, and an RA who picks it up will waste a day
rediscovering that the NSO site is down. It sorts below the real defects and
above the settled tiers.
## The load-bearing part is the probe, not the label
A blocker is a CONDITION THAT CHANGES. A broken website heals; a dead contact
gets replaced. **"We hope we'll find it someday" is not a plan; a scheduled
recheck is.** So `load_blocked()` REFUSES a row with no `probe` URL (it could
never be rechecked, and would silently become the permanent close it explicitly
is not) or no `blocker` (an unexplained block is indistinguishable from
neglect). `bin/coverage_refresh.sbatch` -- which already self-resubmits, is
connectivity-gated, and already opens GitHub issues -- re-probes them.
## Two further bugs this flushed out
**Blockedness is a property of the SOURCE, not of a build.** I first wired it
only into the build-failure branch, so `make matrix-coverage` reported
`blocked=0` while four sources were plainly blocked. We do not need to try and
fail to build Nepal to know NSO's site is down. Hoisted; it now applies to the
cheap coverage-only layer.
**A blocked source we do not MODEL as a wave still has to appear.**
`Albania/1996` has a directory, Documentation, and a catalog id -- but no `_/`
and no `Data/`, so `Country.waves` never enumerates it and every per-wave loop
skipped it in silence. The blocked list is AUTHORITATIVE about sources we want;
it does not defer to whether the model happens to know about them. Otherwise
the denominator quietly omits exactly the waves we most want and cannot get.
## "Countries Without Microdata" is now EMPTY, and that is the point
Both of its entries were misclassified live gaps, each hidden behind a label
that reads "nothing to see here":
Nepal -> `blocked` (above).
Armenia -> NOT MISSING AT ALL. `discover_waves()` defaults to
collection='lsms', but Armenia's **18 waves of Integrated Living
Conditions Survey (2001-2018)** sit under repositoryid='central' --
invisible to our discovery, not absent from the world. Plausibly
the largest actionable acquisition gap in the repo. Filed as
**GH #597**, which also covers South Africa's General Household
Survey 2015-2025 (repositoryid='datafirst', 10 waves).
#597 matters beyond those two countries: we are teaching the matrix to report an
honest denominator of waves the WB publishes that we lack -- and that fix is
INCOMPLETE while `discover_waves` itself can only see one collection. We would
have built a denominator that is confidently wrong.
The ruler was miscalibrated. So was the ruler's own instrument.
Tests: 10 new (blocked beats n/a; probe/blocker refusal; un-modelled wave still
appears; a genuine build failure is still `broken`). 49 passing in
test_coverage_matrix.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 0 was meant to make the denominator honest. It did -- and the honest denominator is far larger than the matrix ever suggested, because the instrument could not see most of it. Armenia (18 ILCS waves, repo=central) and South Africa (10 GHS waves, repo=datafirst) are both IN REMIT (Ethan, 2026-07-12) and both structurally invisible to discover_waves(collection='lsms'). The queue as now known: ~43 waves to ACQUIRE, ~35 already-downloaded waves to CONFIGURE, on top of the 456 absent cells. None of this is new work appearing. It is work that was always there and could not be counted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ources Backlog reconciliation, and a prior-art miss I should own: **GH #552 (June 2026) already audited WB coverage gaps "tiered by obtainability"** — Tier 1 (WB-direct) / Tier 2 (NSO-redirect) / Tier 3 (licensed, fee-gated) — and even cites the Nepal->NSO precedent. I built the `blocked` tier without reading it. CLAUDE.md tells me to run the prior-art-ledger skill at the start of a non-trivial task; I skipped it. The reconciliation was productive in BOTH directions. ## What #552 gives us: `blocker_kind` #552's tiering is the right axis, and it belongs in the machinery as STRUCTURE rather than prose, because the kinds route to completely different people: nso-redirect the WB catalogues it but redirects to a national statistics office whose channel is broken. -> chase a human. fee-gated obtainable, for money and permission (Vietnam GSO: USD 200-2000 + consent). -> a BUDGET decision, not an engineering one. NEVER queue this for an RA. unconfirmed we believe it is blocked but have not proven it; it may be a plain obtainable TODO (Albania 1996). The honest state when you have not actually checked. "The site is down" and "it costs $2,000" are not the same blocker. Flattening them is how a money question silently becomes an engineering backlog item that nobody can ever close. blocked_sources.csv grows from 4 rows to 8, absorbing #552's Tier 2/3: Jamaica (JSLC, STATIN redirect), Ecuador (ECV 1995/1998, INEC redirect), and Vietnam (VLSS/VHLSS x5, GSO, fee-gated). They are now git-tracked, probed, and re-checked — not a snapshot in an issue that ages quietly. ## What we give #552: two of its claims are stale, one is wrong in fact - **"Kyrgyz Republic — wholly-missing"**: NOT missing. 4 waves, 146 DVC sidecars (~378 MB) are already downloaded; only the CONFIG is absent. It is one of the 10 `unconfigured` countries (#593). An add-feature job, not an add-wave one — a materially cheaper class of work than #552 assumes. - **"South Africa: no later WB-LSMS waves"**: true within `collection='lsms'`, FALSE in fact. The General Household Survey 2015-2025 sits under `repositoryid='datafirst'`. Same blind spot hides Armenia's 18 ILCS waves under `repositoryid='central'`. Both now in remit; see GH #597. #552's conclusions are correct GIVEN ITS INSTRUMENT, and its instrument could only see one collection. *A negative result from an uncalibrated instrument is a result about the instrument.* That is the third time that exact lesson has landed in this series -- twice on me. Also closed as verified-fixed during the sweep: #571 (Guatemala), #572 (Serbia), #573 (Panama — its data_scheme.yml landed today in a0ba3e1), #574 (GhanaSPS food_acquired; its sibling #579 stays open — GhanaSPS.sample() still raises). And corrected #587, whose stated hypothesis for its largest bloc (Nigeria's 22 `dropped` cells as "false positives") is refuted: the built `t` values are valid wave labels from the OPPOSITE PP/PH round. They are real half-missing features, not a reporting artefact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g the bookkeeping
Organizing principle: **the GitHub backlog should hold exactly what the coverage
matrix cannot.** Any issue that IS a cell is duplicate bookkeeping -- the
Phase-1 sweep re-derives it, sized and sourced, whether or not an issue exists.
Maintaining both is how a backlog becomes a museum.
the matrix owns WHICH cells are missing / broken / blocked (the WHAT)
the backlog owns WHY a cell is hard, cross-cutting bugs, and everything the
matrix CANNOT SEE
That last clause is not a throwaway. The grader has known blind spots, and they
are exactly where issues earn their keep: #588 (Uganda earnings is sane COLD and
broken WARM -- the matrix is a cold build and structurally cannot see it) and
#589 (currency labels -- it does not check them).
Track what the instrument cannot see. Let the instrument track the rest.
## Severity rule: order by HARM, not effort or age
1 silently wrong the user gets a wrong number and does not know. The only
UNRECOVERABLE class -- it propagates into published work.
2 silently missing the user gets less data than they think.
3 loudly broken it raises. Annoying, harmless. A crash is a gift.
4 untidy noise, dead code, ergonomics.
## Top of the board: #503 (Guyana), and it is not close
`i = [ED, HH]` CONFLATES DISTINCT HOUSEHOLDS. The source is at (ED, HH, SN)
granularity: COVERN.dta has 1502 unique (ED,HH) vs 1807 unique (ED,HH,SN). So
roughly 300 real households are silently merged into others.
That is not missing data. It is WRONG DATA THAT LOOKS FINE -- it builds clean
and passes every sanity check. Every Guyana number we have produced is suspect,
and nothing in the matrix flags it. The purest instance of the session's theme:
`sane` is not `blessed`.
## And #537 is that theme INVERTED
`validate_acquisition_source` has ZERO call sites, while its docstring asserts it
is wired into `_finalize_result`. A validation we believe is running, and is
not: an unevidenced POSITIVE. Same disease, opposite sign. A guarantee that
does not execute is worse than no guarantee, because it is trusted.
## Also
- Wave 4 is DECISIONS, not tickets (#575 purchased-value-only; #226; #279;
#438/#439; Vietnam's USD 200-2000 fee gate). An RA cannot close these by
working harder, and leaving a judgement in an engineering queue is how it
silently becomes a default.
- A large fraction of the ~29 open issues are not defects at all -- they are
cells, decisions, or acquisitions filed as issues because there was nowhere
else to put them. Now there is.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ligon
force-pushed
the
feat/coverage-blocked-tier
branch
from
July 12, 2026 17:07
9e7f955 to
7f80b67
Compare
ligon
added a commit
that referenced
this pull request
Jul 12, 2026
…time Ethan's rulings on #597: South Africa takes the GHS **and** the IES + LCS (28 waves). Each series is now acquired for a STATED reason: GHS (21) roster, housing, education, employment. NO consumption module -- it cannot populate `food_acquired`. IES (5) income + expenditure + acquisition diary. LCS (2) living conditions + expenditure diary. IES and LCS are what feed the demand-estimation path. So the ZAF pin becomes r"_(IHS|GHS|IES|LCS)_" -- 29 rows, zero over-match, against the 320 ZAF rows `datafirst` actually returns. CAPABILITY PRE-POPULATES `not_asked` ------------------------------------ When SA-GHS lands, `South Africa/food_acquired/2015` grades `absent`, someone files it as a gap, and a probe (label sweep + sibling differential + questionnaire) eventually rediscovers a fact the WB catalog told us the day we acquired it. That is archaeology to recover knowledge we already had. `lsms_library/capability.py` records what a *series* measures when it is acquired. A `SeriesCapability` names the features it `provides` and `lacks`; `proposed_absent_verdicts()` turns each `lacks` entry into rows for `.coder/coverage/absent_verdicts.csv` in the schema `load_verdicts()` already reads. The probe sweep then only adjudicates cells inherited WITHOUT a capability record -- a shrinking set, not a permanent tax. AND IT MAY NOT CLOSE A CELL ON A CATALOG BLURB ---------------------------------------------- This is the whole discipline. A capability asserted from a cataloguer's topic list is NOT C4. Closing a cell on it would be the Albania mistake with better paperwork -- an unevidenced negative is unfalsifiable, and therefore permanent whether or not it is true. So each record carries a `validation` level, and the verdict is DERIVED from it rather than chosen by the caller: catalog-only WB topics / abstract -> `unsure` (no close) data-validated C1 label sweep negative -> `unsure` (no close) questionnaire-validated C4: the questionnaire read -> `not-asked` (closes) `data-validated` deliberately does not close either: a negative label sweep is exactly as consistent with `asked-not-distributed` as with `not-asked`. That is precisely why C4 is mandatory. Every record shipped here is `catalog-only`, so every one proposes `unsure`. The 21 GHS `food_acquired` rows it would emit are all `unsure`, `checks_run` empty, evidence prefixed `[catalog-only; C4 NOT run]`. Upgrading a series to `questionnaire-validated` -- one RA, one PDF -- is what converts them into a permanent `not-asked`: a bounded human step per SERIES, not per cell. `capability.audit()` enforces the invariant; `tests/test_capability.py` proves a catalog-only record cannot close a cell *through the real `load_verdicts()`*, and that the questionnaire upgrade does. No verdict rows are written -- SA is not acquired. The mechanism is proven, the shape is up for review. Rebased onto development: #596 resolved the Nepal/Niger/Senegal/Nigeria provenance gaps I had reported, and #598's `blocked` tier supersedes my data_info.yml edit (its emptied "Countries Without Microdata" map is taken wholesale). Census: 79 rows, 54 missing in-remit waves. Refs #597. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The state between "obtainable" and "does not exist": the data exists, we know where it is, and we cannot get it because the acquisition channel is broken.
ALB_1996_EWS) both exist; zero data files.I had this wrong two commits ago, and the mistake is the point
In #593 I graded these
n/a, under "Countries Without Microdata". Butn/ameans "no readiness applies" — it reads as nothing to see here, and it buried a gap we actively want.That is the exact lumping error this module exists to prevent, committed by its author, inside the fix for a different lumping error. Worth stating plainly rather than quietly correcting.
blockedis visible (it counts against us) but not actionable by us — nobody closes it by trying harder, and an RA who picks it up will spend a day rediscovering that the NSO site is down. It sorts below the real defects and above the settled tiers.The load-bearing part is the probe, not the label
A blocker is a condition that changes. A broken website heals; a dead contact gets replaced.
So
load_blocked()refuses a row with:probeURL — it could never be rechecked, and would silently become the permanent close it explicitly is not;blocker— an unexplained block is indistinguishable from neglect.bin/coverage_refresh.sbatchalready self-resubmits, is connectivity-gated, and already opens GitHub issues. It re-probes them. No new loop.Two further bugs this flushed out
Blockedness is a property of the SOURCE, not of a build. I first wired it only into the build-failure branch, so
make matrix-coveragereportedblocked=0while four sources were plainly blocked. We do not need to try and fail to build Nepal to know NSO's site is down. Hoisted — it now applies to the cheap coverage-only layer (no builds, no auth).A blocked source we don't MODEL as a wave still has to appear.
Albania/1996has a directory,Documentation/, and a catalog id — but no_/and noData/, soCountry.wavesnever enumerates it and every per-wave loop skipped it in silence. The blocked list is authoritative about sources we want; it does not defer to whether the model happens to know about them. Otherwise the denominator quietly omits exactly the waves we most want and cannot get."Countries Without Microdata" is now EMPTY — and that is the finding
Both entries were misclassified live gaps, each hidden behind a label reading "nothing to see here":
blocked. Data exists; NSO's site is down.discover_waves()defaults tocollection='lsms'. Armenia's 18 waves of Integrated Living Conditions Survey (2001–2018) sit underrepositoryid='central'— invisible to our discovery, not absent from the world. Plausibly the largest actionable acquisition gap in the repo. Filed as #597, which also covers South Africa's General Household Survey 2015–2025 (repositoryid='datafirst', 10 waves).#597 matters beyond those two countries. We are teaching the matrix to report an honest denominator of waves the WB publishes that we lack (#593, #595) — and that fix is incomplete while
discover_wavesitself can only see one collection. We would have shipped a denominator that is confidently wrong: "no missing waves" for Armenia, while 18 sit in the catalog.The ruler was miscalibrated. So was the ruler's own instrument.
Tests
10 new (
blockedbeatsn/a; probe/blocker refusal; un-modelled wave still appears; a genuine build failure still reportsbroken). 40 passing intest_coverage_matrix.py.🤖 Generated with Claude Code
https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe