Comprehensive Source-Check Expansion: TableBase, FactBase, and WikiBase Verification

[OAGr]

Context

The Source Checks dashboard (E2200) currently shows 725 verdicts across only 3 record types: fact (477), personnel (185), and division (63). Meanwhile, the wiki has ~700 MDX pages, ~1,000 entities, ~3,000 grants, ~1,000 publications, and thousands of other structured records — all with source URLs that could be verified but aren't.

The verification infrastructure is already built (unified source_check_verdicts + source_check_evidence tables, LLM batch checker, dashboard with filters), but it's only being used for a fraction of the data. This discussion proposes expanding source-checking into a comprehensive data integrity system covering all three data layers.

Current State

What Works

• Unified schema: source_check_verdicts uses a polymorphic recordType field — any record type works
• Two-phase verification: Deterministic checks (format, plausibility) + LLM batch verification (Haiku, ~$0.0001/record)
• Dashboard: E2200 already groups/filters by record type dynamically
• Auto-update integration: Contradicted entities are blocked from auto-update (safety gate)
• CLI: crux tb source-check-records --table=<name> runs checks

What's Checked Today

Record Type	Verdicts	Notes
fact (FactBase)	477	Via crux fb source-check — separate system, results in unified table
personnel	185	Via crux tb source-check-records
division	63	Via crux tb source-check-records
Total	725	3 of 15+ checkable types

What's NOT Checked

Record Type	Est. Records	Source Field	Priority	Notes
Wiki pages (MDX)	~700	footnotes, <R> links	CRITICAL	Prose claims, citations, statistics — highest user-facing impact
grants	~3,000	sourceUrl	HIGH	Amounts, dates, grantees verifiable against funder sites
publications	~1,000	url, doi, arxiv_id	HIGH	Year, citation count, authors verifiable via APIs
funding-rounds	~500	source	MEDIUM	Amounts, dates verifiable against press/Crunchbase
investments	~1,000	source	MEDIUM	Amounts, investors verifiable
entity events	~2,000	source	MEDIUM	Dates, descriptions verifiable against news
benchmark results	~500	sourceUrl	MEDIUM	Scores verifiable against leaderboards
funding programs	~200	source	LOW	Deadlines, eligibility verifiable
entity assessments	~500	source	LOW	Ratings with evidence docs
equity positions	~500	—	LOW	No source URL field currently

The Wiki Page Gap (Critical)

Wiki pages are the primary user-facing content and currently have the least systematic verification. There are several existing systems that partially cover this, but they're fragmented:

Existing Wiki Verification Systems (Fragmented)

1. Citation accuracy (citation_quotes table, /internal/citation-accuracy) — Checks individual [^N] footnote quotes against their source URLs. Has quoteVerified, accuracyVerdict, accuracyScore fields. This works but only covers citations that have been explicitly footnoted.

2. Hallucination risk (hallucination_risk_snapshots, /internal/hallucination-risk) — Scores entire pages for unsupported claims. Produces a risk score but doesn't verify individual claims.

3. Citation health (computed at build time) — Counts how many footnotes have been quote-verified. Shown as a banner on wiki pages.

4. Semantic diff (crux/lib/semantic-diff/) — Extracts factual claims before/after page edits and checks for contradictions. Runs during the improve pipeline but doesn't persist verdicts.

What's Missing for Wiki Pages

• Unfootnoted claims: Many pages make factual claims (dates, numbers, comparisons) without footnotes. These are invisible to the citation system.
• Stale content: Pages written in 2024 may reference "current" data that's now outdated. No systematic staleness detection.
• Cross-page consistency: Page A says "founded in 2015" but Page B says "founded in 2016". No cross-reference checking.
• Data-prose agreement: A page's prose says "revenue of $500M" but the FactBase fact or entity data says $450M. No automated detection.
• EntityLink validity: <EntityLink id="X"> components reference entities that may have been renamed, merged, or deleted.

Proposed: Wiki Page Verification Pipeline

Treat each wiki page as a "record" in the source-check system:

• recordType = "wiki-page", recordId = page slug
• Extract claims from prose (dates, numbers, named entities, comparisons)
• Cross-reference against: FactBase facts, entity metadata, citation sources, other wiki pages
• Store verdicts in the unified source_check_verdicts table
• Surface on the Source Checks dashboard alongside TableBase verdicts

Proposed Architecture

Phase 1: Expand TableBase Coverage (Low-Hanging Fruit)

The crux tb source-check-records pipeline already works. Expand it to all registered tables:

1. Update CLI help text to advertise all 9 registered tables (not just 4)
2. Add table-specific deterministic checks for grants, publications, etc.
3. Run initial batch on all tables to establish baseline
4. Estimated cost: ~10,000 records × $0.0001/record = ~$1.00 total

Phase 2: Unify FactBase Source-Checking

Currently crux fb source-check is a separate system that happens to write to the same verdicts table. Unify it:

1. Route FactBase checks through the same crux tb source-check-records --table=fact path
2. Deprecate the separate crux fb source-check command (or make it an alias)
3. Benefit: single pipeline, single dashboard, single recheck scheduler

Phase 3: Wiki Page Verification (Highest Impact)

This is the most ambitious but highest-value phase:

Step 1: Claim extraction

• Parse MDX pages to extract factual claims (using LLM or heuristic extraction)
• Categorize: dates, numbers, comparisons, named entity references, causal claims

Step 2: Cross-reference checking

• Compare extracted claims against FactBase facts, entity metadata, and other wiki pages
• Flag contradictions between prose and structured data
• Detect stale temporal claims ("as of 2024...")

Step 3: Source verification

• For claims with footnotes: verify the footnote supports the claim (existing citation system)
• For claims without footnotes: flag as "unsourced" with severity based on claim type
• For claims referencing external data: fetch and verify

Step 4: Staleness detection

• Track page lastEdited vs. entity lastUpdated vs. auto-update news dates
• Flag pages where significant news has occurred since last edit
• Already partially implemented in auto-update routing

Phase 4: Recheck Scheduling

The nextCheckDue field exists in source_check_verdicts but nothing reads it:

1. Scheduled runner (GitHub Actions or groundskeeper): query verdicts where nextCheckDue < now()
2. Priority queue: contradicted > outdated > partial > unverifiable > confirmed
3. Budget cap: $X/day, process highest-priority first
4. Staleness rules: Recheck annually for confirmed, monthly for partial, weekly for contradicted

Phase 5: Unified Dashboard

The E2200 dashboard already supports this — it groups by recordType dynamically. As new types flow in, they appear automatically. Improvements:

1. Coverage metrics per type: "grants: 2,847 of 3,000 checked (95%)"
2. Trend charts: verdicts over time (are we getting better or worse?)
3. Drill-down to entity: "Show all verdicts for Anthropic across all record types"
4. Action queue: "Top 10 contradicted items to fix today"

Cost Estimates

Phase	Records	Est. Cost	Frequency
Phase 1: TableBase expansion	~10,000	~$1	One-time + monthly recheck
Phase 2: FactBase unification	~3,000	~$0.30	Monthly
Phase 3: Wiki page verification	~700 pages × ~20 claims = ~14,000	~$2-5	Monthly
Phase 4: Recheck scheduling	~5,000/month	~$0.50/month	Ongoing
Total initial run	~27,000	~$5-8	—
Monthly maintenance	~10,000	~$1-2/month	—

Open Questions

1. Should wiki page verdicts show on the page itself? (e.g., "3 claims on this page are contradicted by sources" banner)
2. Should contradicted verdicts block page publication? (Currently auto-update blocks on contradicted entities, but manual edits don't)
3. How to handle "unverifiable" — is it a problem? 33% of current verdicts are unverifiable (source can't confirm or deny). Should we source-upgrade these, or accept them?
4. FactBase vs. WikiBase disagreement: When prose says X but FactBase says Y, which wins? (Currently FactBase is authoritative — should the system auto-flag the wiki page for correction?)
5. Per-field vs. per-record verdicts: The schema supports fieldName-level verdicts but nobody uses them. Worth the granularity?
6. Integration with improve pipeline: Should crux w improve automatically check verdicts before/after editing a page?

Next Steps

Looking for feedback on:

• Priority ordering of the phases
• Whether wiki page verification (Phase 3) should be pulled earlier given its user-facing impact
• Budget/frequency preferences for ongoing verification
• Whether to unify citation-accuracy + hallucination-risk + source-check into one system or keep them separate

[OAGr]

Clarification on wiki page verdicts: We already have infrastructure for showing verification results on wiki pages. See Kalshi (E537) which shows "Citation verification: 42 verified, 7 flagged, 4 unchecked of 87 total" via the CitationHealthBanner component.

The existing citation verification system (citation_quotes table, CitationHealthBanner, CitationOverlay, footnote tooltips with verdict badges) should be migrated to use the unified Source Check system rather than remaining a separate pipeline. This means:

1. Citation verdicts should write to source_check_verdicts with recordType=citation instead of the separate citation_quotes.accuracyVerdict field
2. The CitationHealthBanner should read from the unified verdicts table
3. The Source Checks dashboard (E2200) should show citation verdicts alongside TableBase/FactBase verdicts
4. This eliminates the fragmentation between 4 separate verification systems (citation accuracy, hallucination risk, semantic diff, source-check)

So the answer to the open question "Should verdicts show on wiki pages?" is: they already do for citations — the real task is unifying the backend so all verification flows through one system, not building new UI.

[OAGr]

Source-Check System Audit — Comprehensive Problems & Improvements

Generated 2026-03-25 from 3 parallel review agents (Playwright UX audit, deep code review, data flow audit).

Executive Summary

The source-check system has solid fundamentals — unified schema, working pipeline, multiple display surfaces — but suffers from fragmentation, inconsistency, and missing context. The display layer shows raw internal IDs where names should be, duplicates evidence without deduplication, and lacks claim context. The pipeline sends opaque stableIds to the LLM checker, produces duplicate evidence rows on re-runs, and doesn't set entityId for most record types.

32 code issues found across 14 files. 3 GitHub issues filed during this session (#3136, #3137, #3138).

---

Part 1: Display Layer Issues

1.1 Missing Claim Context (CRITICAL UX)

Problem: The source-check detail page (/source-checks/:type/:id) shows verdict and evidence but doesn't clearly state what claim is being verified.

• For facts: Now shows "Entity — Property: Value" (fixed in this PR)
• For personnel: Shows "Person @ Org" but not what's being checked (role? dates? affiliation?)
• For divisions: Shows division name but not what's being verified

Impact: Users see "confirmed 95%" but have to read the full reasoning text to understand what was confirmed.

Fix: The detail page should show a structured "Claim" section at the top:

• Fact: "Brookings Institution founding date = 1916"
• Personnel: "Jaime Sevilla is a Researcher at Epoch AI (since 2021)"
• Division: "Safety Systems is a division of Anthropic, status: active"

1.2 Duplicate Evidence Rows (HIGH)

Problem: Re-running source checks against the same record produces new evidence rows without removing old ones. Example: f_Tc2pW7xR4a has 5 evidence rows, all from en.wikipedia.org, 4 with identical notes.

Filed: #3136

Partial fix in this PR: Evidence is now grouped by source URL and deduplicated in the display layer (same verdict + similar notes collapsed with "X similar checks" badge). But the underlying data still has duplicates.

1.3 Raw Internal IDs Shown to Users (HIGH)

Problem: Multiple surfaces show opaque IDs:

• Entity column shows "-" for personnel/division (entityId not set in verdicts) — Filed Source-check verdicts missing entityId for personnel and division records #3138
• Record column shows raw stableIds when name resolution fails
• "Fact f_Tc2pW7xR4a" as page title (now partially fixed with claim summary)

1.4 Three Different Display Surfaces, Three Different UX (MEDIUM)

Feature	/source-checks list	E2200 dashboard	Detail page
Entity column	✅ (with links)	✅ (with links)	Partial (entity link in header)
Name resolution	Server-side (SSR)	Client-side (useEffect)	Server-side (SSR)
Verdict sorting	Server-side (SQL)	Client-side (VERDICT_PRIORITY)	N/A (single record)
Search	Client-side (on page of 50)	Client-side (on all loaded)	N/A
Evidence detail	Click → detail page	Expand inline	Full display
Stat cards	4 cards	6 cards	Verdict card

Impact: Users see different information and behavior depending on which surface they use.

1.5 Truncation Length Inconsistencies (MEDIUM)

What	/source-checks table	E2200 viewer
Entity name	28 chars	28 chars ✓
Entity ID fallback	12 chars	12 chars ✓
Record name	40 chars	30 chars ✗
Record ID fallback	20 chars	15 chars ✗

1.6 Checker Model Shown as Raw String (LOW)

• E2200 expanded detail: Now uses formatCheckerModel() (fixed in this PR)
• /source-checks list: Doesn't show model
• Detail page: Uses formatCheckerModel() (fixed in this PR)

---

Part 2: Pipeline Issues

2.1 LLM Checker Receives Raw StableIds (HIGH)

Problem: The source-check prompt builder sends raw internal stableIds (e.g., "ULjDXpSLCI") as part of the record data. The LLM correctly flags these as unverifiable.

Filed: #3137

Fix: crux/tablebase/source-check.ts prompt builder should resolve entity references to human-readable names before constructing the verification request.

2.2 entityId Not Set for Personnel/Division Verdicts (HIGH)

Problem: When submitting verdicts, the source-check runner doesn't set entityId for personnel or division records. This means the Entity column is empty for 248 of 725 verdicts (34%).

Filed: #3138

Fix: For personnel, set entityId = orgEntityId or personEntityId. For divisions, set entityId = organizationId.

2.3 No Deduplication on Re-runs (MEDIUM)

Problem: Running source checks again produces new evidence rows. The source_check_evidence table has no unique constraint on (recordType, recordId, sourceUrl, checkerModel).

Filed: #3136

2.4 Stale Checker Models in Evidence (MEDIUM)

Problem: Old evidence rows use claude-3-haiku, newer ones use claude-haiku-4-5-20251001. The nextCheckDue field exists in the schema but nothing reads it — no recheck scheduling.

2.5 Only 3 of 15+ Record Types Checked (LOW — already tracked)

Covered in Discussion #3119.

---

Part 3: Architectural Issues

3.1 Three Separate Name Resolution Paths (MEDIUM)

1. Main page (page.tsx): SSR, uses wiki-server API for records + local database for entities
2. E2200 dashboard (entity-source-checks-viewer.tsx): Client-side, uses proxy API for all types
3. Wiki-server (source-checks.ts): Implements full resolution logic

If resolution logic changes for a record type, it must be updated in 3 places.

Fix: Consolidate to one path — all resolution through the wiki-server API, with the proxy handling entity resolution locally as a fast path.

3.2 Client-Side Search on Server-Fetched Pages (MEDIUM)

The /source-checks page fetches 50 verdicts per page (server-side), then does client-side search. This means:

• "2 of 500 results" after searching — misleading (only 50 were fetched)
• Search only works within the current page, not across all verdicts

Fix: Move search to server-side (add q param to the wiki-server verdicts endpoint).

3.3 Duplicate VerdictBadge Definitions (LOW — partially fixed)

Was 3 copies, now 2 (E2200 viewer imports from shared). The factbase-source-checks-table.tsx still has its own copy.

3.4 Verdict Priority Defined in 3 Places (LOW — partially fixed)

• source-checks-shared.tsx: 0-5 scale (now shared)
• source-checks.ts (wiki-server): 100-0 scale (SQL CASE)
• E2200 viewer: Now imports from shared ✓

---

Part 4: Missing Features

4.1 No Recheck Scheduling

The nextCheckDue and needsRecheck fields exist but nothing reads them. No GitHub Action or groundskeeper task rechecks stale verdicts.

4.2 No Confidence Threshold Filter

Users can filter by verdict type but not by confidence range (e.g., "show only <50% confidence").

4.3 No Bulk Operations

No way to "recheck all contradicted verdicts" or "clear all outdated evidence" from the UI.

4.4 No Revision History

No way to see how a verdict changed over time (was it confirmed, then contradicted after a recheck?).

4.5 No Export

No CSV/JSON export for external analysis.

4.6 No "Needs Recheck" Filter

The stat card shows the count but there's no filter button to show only items needing recheck.

---

Part 5: Security & Validation

5.1 No Rate Limiting on Proxy APIs (LOW)

The 3 proxy routes don't implement rate limiting. A malicious client could flood the wiki-server.

5.2 Parameter Naming Inconsistency (LOW)

• verification-names-proxy: record_type (snake_case)
• verification-detail: recordType (camelCase)

---

Issues Filed This Session

#	Title	Priority
#3136	Deduplicate re-runs and flag stale checker models	MEDIUM
#3137	LLM prompt sends raw stableIds instead of names	HIGH
#3138	Verdicts missing entityId for personnel/division	HIGH

Discussion Filed

#	Title
#3119	Comprehensive Source-Check Expansion: TableBase, FactBase, WikiBase

Changes Shipped in This PR (#3135)

Change	Impact
Shared VerdictBadge (removed 1 of 3 duplicates)	Consistency
Shared VERDICT_PRIORITY	DRY
formatCheckerModel()	"Haiku 4.5" instead of raw model string
Evidence grouped by source URL	Reduces visual noise
Evidence deduplicated in display	Collapses 5 identical cards → 1
Fact claim summary in header	"Entity — Property: Value"
Inline verdict on fact detail page	Badge + confidence, not just a link
Entity column on /source-checks list	Resolved names with wiki page links
Record links via getRecordHref for all types	Facts, grants, divisions, etc.
Contradicted/Outdated stat cards	Replace low-value "Needs Recheck: 0"
robots: { index: false } on detail pages	SEO

[OAGr]

Playwright UX Audit — 63 Issues Found

Full audit document: see .claude/audit/source-check-ux-audit.md in the repo.

Critical Issues (fix immediately)

1. ~110 of 185 personnel records show "Unknown" — the name resolver returns "Unknown @ Epoch AI" instead of actual person names. The personnel source-check feature is nearly useless.
2. Raw numeric IDs as names — "304 @ Anthropic" (should be Sam McCandlish), "187 @ 263" (completely opaque)
3. Raw internal stableIds as org names — "Unknown @ FISYNGdZcf4n", "Unknown @ 4c2Xd10VJDYX"
4. new: prefix leaked to display — ~15 Apollo Research records show "new:Annika Hallensleben @ Apollo Research"

High Priority

5. Entity column is "-" for ALL 248 personnel + division rows (34% of data)
6. Summary stats don't update with filters — header always shows global totals regardless of active type/verdict filter
7. Verdict filter counts are global, not contextual — selecting "Fact (477)" type still shows "contradicted (40)" globally instead of contradicted facts only
8. Action Queue items are NOT clickable — shows 40 items needing attention but none are links
9. "Verified: 6%" label is misleading — actually means "coverage" or "checked", not "passed verification"
10. No breadcrumb on /source-checks list page — orphaned in the site hierarchy

Data Quality Issues Found in Live Data

• Personnel "304 @ Anthropic" is Sam McCandlish (raw personEntityId used as name)
• "187 @ 263" — both person AND org are raw numeric IDs
• Contradicted count mismatch: header says 20 but list shows 40
• Evidence text contains raw stableIds ("ENl8sgChDQ", "mK9pX3rQ7n") from LLM seeing opaque IDs

Issues by Severity

Severity	Count
Critical	4
High	10
Medium	16
Low	10
Total	63 (some issues span multiple categories)

[OAGr]

Connection to Discussion #2950 (Unified Verification Architecture)

The current implementation is a direct descendant of the design in #2950:

#2950 Design	Current Implementation
verification_evidence	source_check_evidence
verification_verdicts	source_check_verdicts
formatClaim(record_type, record_id, field_name)	Partially built for facts only (claim summary in detail page header)
Cell-level + row-level verification	Schema supports it (fieldName column) but not used in practice
Per-entity verification page	Not built yet — have per-record pages at /source-checks/:type/:id

What's missing from the #2950 vision

1. formatClaim() for all record types — Unified Verification Architecture — Data Layer & Verification Decisions (March 2026) #2950 specified dynamic claim formatting like "Organization [Anthropic] as [revenue 2025] of [\M]". Currently only facts have claim summaries. Personnel/division/grant detail pages show raw names without the claim structure.

2. Cell-level verification — the schema has fieldName but nothing uses it. Grants should verify amount and date independently per Unified Verification Architecture — Data Layer & Verification Decisions (March 2026) #2950.

3. Per-entity verification summary — Unified Verification Architecture — Data Layer & Verification Decisions (March 2026) #2950 proposed /verify/[entity] showing all verdicts grouped by entity. Currently you can only see per-record detail or the flat dashboard list.

4. Recheck scheduling — nextCheckDue and needsRecheck fields exist but no runner reads them.

5. Automation budget controls — Unified Verification Architecture — Data Layer & Verification Decisions (March 2026) #2950 mentioned crux verify <entity> with budget caps. The source-check runner exists but without budget controls.

Recommended next steps (based on #2950)

1. Build formatClaim() as a shared server-side utility that works for all record types
2. Add per-entity verification summary page (group all verdicts for one entity)
3. Implement cell-level verification for grants and investments
4. Build the recheck scheduler (GitHub Action or groundskeeper task)

[OAGr]

Integration Audit: Source-Checks Are More Connected Than Expected

Investigation revealed the source-check system is already widely integrated:

Already showing verification badges (10+ locations)

• Wiki pages: VerificationStatus component shows entity-level verdict summary (confirmed count, avg confidence, expandable detail table)
• SourceCheckBadge component used on: org equity sections, individual grant pages, funding programs, people career history, divisions table, funding rounds, investments, legislation stakeholders
• Main navigation: /source-checks linked under Data dropdown
• FactBase facts: Inline verdict badge with confidence + detail link

Gaps remaining

• Org/people directory cards don't show verification badges
• Wiki data tab (/wiki/E<N>/data) doesn't show source-check summary
• DataInfoBox sidebar doesn't show verification status
• Two separate systems coexist: citation verification (CitationHealthBanner) and source-check (VerificationStatus)

Key finding: SourceCheckBadge supports 9 record types

grant, personnel, division, funding-program, funding-round, investment, equity-position, policy-stakeholder, division-personnel. This is more than the 3 types (fact, personnel, division) currently in the source-check verdicts table.

[OAGr]

Data Quality Audit: Critical Pipeline Findings

Adversarial review of live source-check data revealed systemic quality issues:

~70% False Positive Rate in Contradictions

Of 40 contradicted verdicts, only ~10-12 are genuine errors. Three LLM failure modes:

1. Range-vs-point: Source says "51-200 employees", wiki says 91 (within range) -> LLM calls it contradicted
2. Temporal mismatch: Source discusses 2025 projections, wiki claims 2024 actuals -> LLM conflates
3. Wrong source: Checking Anthropic.com for evidence of prior OpenAI employment -> obviously not found

Verdict Aggregation Bug

Stale contradictions persist after re-checks confirm data. Example: Elon Musk net worth has a recent "confirmed" check but overall verdict still shows "contradicted" from older check. The aggregation uses "worst verdict wins" instead of "most recent verdict wins".

Web Scraping Quality

Many "unverifiable" verdicts stem from the scraper returning nav/menu HTML instead of page body content (CSIS Wadhwani Center, ForecastBench). This is a systematic fetching problem.

Personnel Name Resolution Still Broken (~60%)

Despite wiki-server improvements, ~60% of personnel records show raw hash IDs (e.g., "boXCNeZK3w @ Epoch AI" = Maria de la Lama). Entity column is "-" for ALL 185 personnel.

Coverage: 94% Unchecked

Only 805 of 12,439 records (6%) have any source checks. Largest gaps: grants (5,831), citations (3,490), wiki pages (835).

Issues Filed

• Verdict aggregation bug (stale contradictions persist)
• ~70% false positive rate in contradicted verdicts

[OAGr]

Status (April 2026): Partially done. Source-check orchestration exists (crux source-check orchestrate), unified tables (source_check_evidence + source_check_verdicts) are in production. FactBase fact-checking and personnel checks work. Coverage is still low: 94% of TableBase records have zero coverage per #3161. WikiBase prose checking and bulk automated sweeps are not yet built.

[OAGr]

Decomposed into 4 GitHub Issues

Broke this discussion down into concrete, implementable issues based on what exists vs what's missing:

Issues Created

#	Title	Phase	Effort
#3657	Integrate wiki-page source-checking into the orchestrator pipeline	Phase 3 (WikiBase)	Medium — wiring existing source-check-wiki-pages logic into the orchestrator's item collection and dispatch
#3661	Add automated recheck scheduling via GitHub Actions	Phase 4 (Recheck)	Small — GitHub Actions workflow + groundskeeper task, command already exists
#3667	Run source-check backfill across all TableBase record types	Phase 1 (TableBase)	Small — execution, not code. Orchestrator already supports all 13 record types
#3671	Migrate citation verification pipeline to unified source_check_verdicts table	Phase 2 (Unification)	Large — dual-write migration, read path switch, depends on #3447

Already Covered by Existing Issues

Several concerns from the discussion are tracked elsewhere:

• Verdict vocabulary unification: Unify verdict vocabularies: source-check vs citation pipeline #3447
• Unverifiable record improvement: Improve source-check coverage for 869 unverifiable records #3609, Auto-suggest better source URLs when source-check returns unverifiable #3548
• JS-rendered page fetching: Use Playwright JS rendering for source-check on JS-heavy pages #3546
• Verification dots on more tables: Extend verification dots to other PG tables (investments, publications, funding-rounds) #3444
• Contradicted verdict triage: Triage 62 contradicted record verdicts from source-check backfill #3607
• Recheck evidence storage bug: source-check-recheck: bad_request when storing evidence for 29/33 items #3472

Recommended Execution Order

1. Run source-check backfill across all TableBase record types #3667 (backfill) — immediate, costs ~$2-5, no code changes needed
2. Integrate wiki-page source-checking into the orchestrator pipeline #3657 (wiki-page integration) — brings the highest-impact content type into the orchestrator
3. Add automated recheck scheduling via GitHub Actions #3661 (recheck scheduling) — makes verification self-sustaining (depends on source-check-recheck: bad_request when storing evidence for 29/33 items #3472 fix)
4. Migrate citation verification pipeline to unified source_check_verdicts table #3671 (citation unification) — largest scope, depends on Unify verdict vocabularies: source-check vs citation pipeline #3447, eliminates the last fragmented pipeline

[OAGr]

Superseded by Discussion #3993 (Source-Check → Sourcing Rename — Implementation Plan).

The architecture and naming decisions in this discussion have been consolidated into the 7-phase sourcing rename plan. Key changes: routes become /sourcing, tables rename to record_sources/source_verdicts, components standardize on SourcingDot/SourcingBadge.

Phase 0: #4001 | Phase 1: #4002

[OAGr]

Update 2026-04-10: Phase 6 addition — unattended per-org curation runner

Context from the Anthropic pilot (Linear milestone "Anthropic → 100% green"): the manual playbook of "for one org, fix unverifiable records by swapping in better source URLs, re-verify, iterate" took Anthropic personnel from 41% → 100% verified-or-better. But this required close human oversight and doesn't scale to the hundreds of orgs on the wiki.

What Phases 1–5 cover vs. what they don't

The existing 5-phase plan in this discussion covers:

• Phase 1: Expand TableBase coverage to more record types — ✅ in progress
• Phase 2: Unify FactBase source-checking — ✅ schema unified, command consolidation pending
• Phase 3: Wiki page verification — 🟡 partial (citation accuracy + hallucination risk exist, but claim extraction doesn't)
• Phase 4: Recheck scheduling — ✅ QUA-71 done (GitHub Action + crux source-check-recheck)
• Phase 5: Unified dashboard — ✅ E2200 exists, coverage metrics per type pending

The missing layer: none of these phases describe the proactive, per-org curation loop that actually drives "confirmed+partial → 100%" for an entity. Phases 1–5 are reactive infrastructure (tables, schemas, schedulers, dashboards). The thing that takes an org and fixes its unverifiable records by finding better sources is not scoped anywhere.

What the Anthropic pilot showed

The manual loop per org is:

1. Query per-entity records, identify unverifiable + partial + unchecked
2. For each, find a better source URL (research step)
3. Write the new URL + reset verdict + register URL as a resource + wait for ingest
4. Run verify-orchestrate --entity=X and measure the delta
5. Iterate on remaining failures until budget or convergence

Per-org cost: ~$0.25 for a 25-record org. Fully automatable once QUA-210 (agentic source-discover v2) and QUA-218 (fix auto-enqueue blind spot — shipped as PR #4113) land.

Proposed Phase 6: Fleet curation runner

A new crux flagship-curate command that:

• Picks the next N orgs by a priority function (importance × staleness × gap-size)
• For each, runs the per-org loop unattended with a budget cap
• Writes per-org delta reports to a flagship_curation_runs table
• Surfaces a fleet leaderboard dashboard
• Files GitHub issues automatically for orgs with 3 consecutive no-delta runs (systemic blockers)
• GitHub Action runs daily with ~$3-5/day budget

Target: fleet-wide confirmed+partial rate improves ≥3% per daily run, converging toward 95%+ for the top 100 orgs within a quarter.

Tracked as: QUA-220 — crux flagship-curate: unattended per-org curation runner.

Dependencies

• QUA-210 (agentic source-discover v2) — the core engine. Current source-discover is a single-shot Exa search per person; the fleet runner needs the batched claims-first version.
• QUA-218 (self-healing ingest auto-enqueue blind spot) — shipped as PR Fix source-fetcher auto-enqueue blind spot for brand-new URLs #4113. Without it, new URLs stall forever and the loop doesn't converge.
• QUA-191 (SOURCE_CHECK_EXEMPT_TYPES) — exempt "direct-source" data classes (prediction markets, arXiv, vendor RSS) so the runner doesn't waste cycles on them.
• QUA-194 (per-org completeness score) — needed for the priority function and the "skip if already converged" check.

Open question for Phase 6

Should the fleet runner also touch wiki pages (Phase 3 territory), or stay strictly on TableBase records? Pilot suggests starting with TableBase (grants, personnel, funding-rounds, divisions, investments) because (a) the verification is deterministic for many fields, (b) per-entity filtering is already supported in verify-orchestrate, and (c) wiki-page claim extraction is a much harder research problem. Phase 3 (wiki-page verification) can layer on top once it exists.

Status update on Phases 1–5

• Phase 1 (TableBase expansion): in progress via QUA-126, QUA-72, QUA-83. Anthropic pilot used verify-orchestrate --entity=X --type=record --table=personnel successfully — the per-entity path works.
• Phase 2 (FactBase unification): still separate crux fb source-check command exists; unification pending.
• Phase 3 (wiki page verification): no new movement; citation accuracy + hallucination risk remain fragmented. Proposed unification is still open.
• Phase 4 (recheck scheduling): ✅ shipped via QUA-71. GitHub Action cron runs weekly.
• Phase 5 (unified dashboard): E2200 works for raw verdict browsing; QUA-194 extends Entity Profile E1929 with per-org completeness scoring (not yet implemented).