Source-Check Coverage Strategy: Closing the Verification Gap

[OAGr]

Context

Source-check verification dots now appear across all directory pages (personnel, grants, divisions, funding programs, funding rounds, investments). But the actual coverage is spotty:

Current state (2026-04-03)

Global source-check stats:

Record type	Checked	Total (est.)	Coverage
Personnel	411	501	~82%
Grant	261	?	?
Division	91	?	?
Investment	9	?	low
Publication	143	?	?
Fact (FactBase)	1,444	?	?
Citation	605	?	?

Per-org example (Anthropic): Only 5 of 27 personnel records have been checked (19%). All 5 came back "unverifiable."

Verdict distribution across all 3,044 checks:

• Confirmed: 1,420 (47%)
• Unverifiable: 927 (30%)
• Partial: 565 (19%)
• Contradicted: 113 (4%)
• Outdated: 19 (<1%)

Related open issues

• Run source-check backfill across all TableBase record types #3667 — Run source-check backfill across all TableBase record types
• Improve source-check coverage for 869 unverifiable records #3609 — Improve coverage for 869 unverifiable records
• Auto-suggest better source URLs when source-check returns unverifiable #3548 — Auto-suggest better source URLs for unverifiable records
• Add Wayback Machine fallback for 404'd source URLs in source-check pipeline #3664 — Wayback Machine fallback for 404d sources
• Use Playwright JS rendering for source-check on JS-heavy pages #3546 — Playwright JS rendering for JS-heavy source pages
• Add automated recheck scheduling via GitHub Actions #3661 — Automated recheck scheduling via GitHub Actions
• Extend deterministic row-matching verification beyond grants to personnel and investments #3669 — Extend deterministic row-matching beyond grants

Questions to decide

1. Priority ordering: Should we backfill unchecked records first (Run source-check backfill across all TableBase record types #3667) or improve unverifiable ones (Improve source-check coverage for 869 unverifiable records #3609)? Backfilling gives users coverage expectations; improving unverifiable gives better data quality.

2. Automation cadence: Should source-checks run on a schedule (daily/weekly via GitHub Actions, Add automated recheck scheduling via GitHub Actions #3661), on record creation, or manually? The batch API is cheap (50% discount) but still has cost.

3. Target coverage: What is the minimum acceptable coverage before we show verification dots on a page? Showing dots when only 5/27 records are checked is arguably worse than showing nothing — it implies the unchecked records are fine (especially given RecordVerificationDot: no visual distinction between 'never checked' and 'unverifiable' #3740, where "never checked" and "unverifiable" look identical).

4. Source URL quality: 30% of checked records come back "unverifiable." Is the problem bad source URLs, paywalled content, or records that genuinely cannot be verified? Should we invest in better source discovery (Auto-suggest better source URLs when source-check returns unverifiable #3548) before running more checks?

5. Cost budget: At current Haiku pricing, what is the estimated cost to reach 95%+ coverage across all record types? Is this a one-time backfill or an ongoing expense?

6. Display threshold: Should we hide verification dots for a record type on an org page until some minimum % of that org records have been checked? E.g., dont show personnel dots unless >=80% of this orgs personnel have been checked.

[OAGr]

Updated coverage data (actual totals)

The gap is much worse than initially estimated:

Record type	Total records	Source-checked	Coverage
Grants	5,904	261	4%
Personnel	501	411	82%
Divisions	112	91	81%
Funding programs	61	0	0%
Funding rounds	91	0	0%
Investments	72	9	12%
Total	6,741	772	11%

Grants are the biggest gap by far — 5,904 records with only 4% checked. Funding programs and funding rounds have zero coverage. The Coefficient Giving funding tab shows this clearly: every single grant row shows an unverified gray dot.

This changes the cost/strategy discussion significantly. At ~5,900 unchecked grants, the batch API cost for a full backfill is non-trivial.

[OAGr]

Session results (2026-04-03)

Coverage before/after

Record type	Before	After	Change
Grants	261 (4%)	5,576 (94%)	+5,315
Personnel	411 (82%)	531	+120
Investment	9 (12%)	10 (14%)	+1
Publication	143 (77%)	144	+1
Total	3,044	11,071	+8,027

What was done

1. Backfilled 5,315 grant verifications using deterministic row-matching against source snapshots. Cost: $0 (no LLM needed for 92% of grants).
2. Fixed pagination clamping (PR fix: clamp pagination limit instead of rejecting (unblocks source-check backfill) #3749, merged). Server now clamps limit > MAX_PAGE_SIZE instead of rejecting.
3. Added rate-limit retry to apiRequest (PR fix: clamp pagination limit instead of rejecting (unblocks source-check backfill) #3749, merged).
4. Changed source-check cron to daily (PR fix: clamp pagination limit instead of rejecting (unblocks source-check backfill) #3749, merged).
5. Fixed received grants verification dots (PR feat: show verification dots on received grants #3794, open). Added ?role=grantee to grants API.
6. Uploaded Manifund snapshot (20K projects via Playwright).

Remaining gaps

All remaining gaps are not_cached (source content not fetched yet):

• 377 grants (Manifund + misc)
• 339 personnel
• 83 divisions
• 63 investments

The daily cron will process these via LLM fallback (~$0.002/record).

Data issues found

• Grant contradiction: Lightcone Infrastructure / Lighthaven grantee mismatch
• Division contradiction: Anthropic "AI Welfare Research" not found in source

[OAGr]

Status Update (2026-04-04)

PRs merged since last session update

Since the Apr 3 session results, 10+ related PRs have landed:

Direct source-check improvements:

• feat: show verification dots on received grants #3794 — Show verification dots on received grants (was open in last update, now merged)
• fix: visually distinguish 'never checked' from 'unverifiable' verification dots #3774 — Visually distinguish "never checked" from "unverifiable" dots (closes RecordVerificationDot: no visual distinction between 'never checked' and 'unverifiable' #3740)
• feat: add verification dots to FactBase components and org profile sections #3816 — Add verification dots to FactBase components and org profile sections
• Add page_snapshots table and fetch pipeline for website sources #3788 — Add page_snapshots table and fetch pipeline for website sources

Display/rendering fixes:

• Fix source-check evidence display and add render audit regression tests #3840 — Fix source-check evidence display + render audit regression tests
• simplify: use getSourceCheckHref(), add sr-only label, remove comment #3834 — Simplify: use getSourceCheckHref(), add sr-only label
• audit: split orchestrate, pagination gate, expand test coverage #3833 — Split orchestrate, pagination gate, expand test coverage
• Fix rendering bugs: org table formatters, source-check JSON, component tests #3787 — Fix rendering bugs: org table formatters, source-check JSON
• fix: bundle citationQuotes by page slug for citation dots #3765 — Bundle citationQuotes by page slug for citation dots

Issue tracker

Issue	Status	Notes
#3740	Closed	Fixed by #3774 — dots now visually distinct
#3664	Closed	Wayback Machine fallback implemented
#3667	Open	Backfill across all TableBase types — grants done (94%), personnel/divisions/investments remain
#3669	Open	Extend deterministic row-matching beyond grants
#3661	Open	Automated recheck scheduling — daily cron now running (set up in #3749)
#3609	Open	Improve unverifiable records (30% of checks)
#3548	Open	Auto-suggest better source URLs
#3546	Open	Playwright JS rendering for JS-heavy sources

Assessment

The big win from this strategy was the grant backfill (4% → 94% at zero LLM cost via deterministic row-matching). Verification dot UX has improved significantly: #3774 fixed the misleading display where "never checked" looked identical to "unverifiable", #3816 extended dots to FactBase facts, and #3788 laid groundwork for better source caching with a proper page_snapshots table.

Answers emerging for original questions:

• Q1 (priority ordering): Backfill-first proved correct — 5,315 grants verified at $0 LLM cost
• Q3 (display threshold): Partially addressed — dots now honestly show "never checked" vs "unverifiable" (fix: visually distinguish 'never checked' from 'unverifiable' verification dots #3774), but the question of hiding dots below a coverage % threshold is still open
• Q4 (source URL quality): Infrastructure support via Add page_snapshots table and fetch pipeline for website sources #3788 (page snapshots) and Add Wayback Machine fallback for 404'd source URLs in source-check pipeline #3664 (Wayback fallback), but no systematic analysis of the 30% unverifiable rate yet
• Q5 (cost budget): Validated cheap — grant backfill was $0, remaining ~862 records at ~$0.002/record ≈ $1.72 total

Next priorities:

1. Extend deterministic row-matching to personnel and investments (Extend deterministic row-matching verification beyond grants to personnel and investments #3669) — same zero-cost pattern
2. Daily cron processing the remaining ~862 uncached records
3. Analyze the 30% unverifiable rate to decide if source URL improvement (Auto-suggest better source URLs when source-check returns unverifiable #3548) is worth investing in

[OAGr]

Branch: claude/deploy-tasks-verify-cmd

Closing — the strategic questions raised here are now answered in #3958 (Source-Check Quality Push), which is the active enforcement plan. #4017 is the umbrella data integrity epic. — Discussion review 2026-04-08.