Source Check System: Audit Findings & Fix Proposals

[OAGr]

Problem Statement

The Source Check system has significant usability and data quality issues that undermine its value as a verification tool. An audit of the production data and codebase reveals six interrelated problems.

Production Data Snapshot (March 2026)

FactBase facts checked (1,444 of 2,417):

Verdict	Count	% of Checked
Confirmed	728	50.4%
Unverifiable	464	32.1%
Partial	198	13.7%
Contradicted	45	3.1%
Outdated	9	0.6%

Other record types: Personnel 81% checked, Divisions 81%, Publications 77%, Investments 78%, Grants 4.4%, Benchmarks 0%, Wiki pages 0%, Citations 0%.

Source URL coverage in YAML: 70.9% of facts have a source URL. 29.1% don't (mostly description and notable-for properties). 38% of source URLs come from Wikidata/Wikipedia.

---

Problem 1: "Avg Confidence" is a meaningless headline stat

The dashboard shows "89% confidence" as a headline number. This is the LLM's self-reported certainty about its own verdict, not an accuracy rate. High confidence on an "unverifiable" verdict means "I'm very sure I can't verify this" — the opposite of useful. Averaging across verdict types makes it doubly meaningless.

What users actually want to know: "How much of our data is verified correct?" and "How much is wrong?"

---

Problem 2: "Unverifiable" is a bucket for 4+ different problems

32% of checked facts are "unverifiable," but this conflates distinct situations requiring different remediations:

1. Dead link (404/gone) — URL no longer exists, needs replacement
2. Topic mismatch — URL about layoffs used to source revenue figures (wrong source cited)
3. Paywall/access blocked — content exists but can't be fetched
4. No source URL — the fact has no source field at all (609 facts, 29% of total)
5. Content too vague — source exists but doesn't contain the specific data point

Each needs different action. Currently they're all lumped together with no way to filter or prioritize.

---

Problem 3: Source URLs don't link to Resources

The source_check_evidence table has a resourceId FK column. The API accepts it. The frontend (FootnoteTooltip.tsx) already renders "View source details" links when resourceId is present. But storeSourceCheckEvidence() in verdict-handler.ts never calls lookupResourceByUrl(). The entire infrastructure is wired up except for one 5-line change.

The claim-verification handler even has resourceId in its job params but doesn't pass it through.

Source check pages show raw URLs as external links instead of linking to the rich Resource pages that already exist for many of these sources.

---

Problem 4: No broken link detection

The source-fetcher reads from the citation_content cache. If a URL isn't cached, it enqueues a resource-ingest job (fire-and-forget) and returns not_cached. There's no feedback loop — the checker never learns whether the URL was dead.

The citation_content table DOES track fetchStatus (200, 404, etc.), but source checks don't query it. Dead URLs just silently become "unverifiable" verdicts with no differentiation from other causes.

---

Problem 5: Claims/entities are confusing in the dashboard

The Verdicts tab shows:

• Record type (e.g., "fact") — too generic
• Entity ID (resolved to name via API, but sometimes fails → shows raw ID)
• Record ID (opaque, e.g., "f_qR5tY9wE1a")
• Reasoning (the LLM's reasoning, NOT the actual claim being checked)

A user looking at this table can't tell what claim was checked or why it matters. There's no link back to the entity page where the fact appears.

---

Problem 6: Dashboard stats hide the real picture

The headline stats ("Total Records", "Checked %", "Avg Confidence", "Contradicted", "Needs Recheck") don't answer the fundamental question: "Should I trust the data on this wiki?"

There's no breakdown of:

• How many verdicts are actually "confirmed" vs "unverifiable"
• What % of source URLs are dead/broken
• Which entity types have the worst coverage
• What % of facts have no source URL at all

---

Proposed Fixes

Fix 1: Resource Linking (Small effort, no dependencies)

In storeSourceCheckEvidence(), auto-lookup resourceId via lookupResourceByUrl() when sourceUrl is provided:

// In verdict-handler.ts:
let resolvedResourceId = params.resourceId ?? null;
if (!resolvedResourceId && params.sourceUrl) {
  const resource = await lookupResourceByUrl(params.sourceUrl);
  if (resource.ok) resolvedResourceId = resource.data.id;
}

Then update the source-check detail page to show "View resource details" links (same pattern as FootnoteTooltip). Run a one-time backfill for existing evidence.

Fix 2: Headline Stat Redesign (Medium effort, no dependencies)

Replace "Avg Confidence" with stats that answer the actual question:

Row 1 — Data Quality:

• Verified Correct: 728 (50.4%) — green
• Has Issues: 252 (17.4%) — amber/red breakdown
• Can't Verify: 464 (32.1%) — gray
• Not Yet Checked: 973 — light outline

Row 2 — Action Items:

• Wrong (fix now): 45 contradicted — red
• Outdated: 9 — amber
• No Source URL: 609 — need source research
• Dead Links: N — need detection

Key insight: separate data quality (of what we've checked) from data coverage (how much we've checked). "Accuracy rate" should be Confirmed / (Confirmed + Contradicted + Outdated), excluding unverifiable — that's a coverage issue, not a quality issue.

Confidence should move to a "System Health" tab, shown per-verdict-type.

Fix 3: Claims Display Improvement (Medium effort, no dependencies)

Restructure the Verdicts table columns:

Entity	Claim	Verdict	Conf.	Source	Checked
Anthropic	revenue = $100M (2023-12)	Contradicted	92%	sacra.com	3d ago
OpenAI	founded = 2015-12-11	Confirmed	95%	wikidata.org	1w ago

• Entity = resolved name, linked to entity page
• Claim = {property} = {expectedValue} (asOf) — the actual thing being checked
• Source = hostname, linked to Resource page (if matched) or raw URL
• Uses expectedValue already present in evidence records

Fix 4: Broken Link Detection (Medium effort, depends on source-fetcher)

Step 1: Before calling the LLM, query citation_content for the URL's fetchStatus. If 4xx/5xx, skip the LLM call and store verdict: unverifiable + fetchStatus: dead_link. Saves money and provides better data.

Step 2: Add dashboard filter for "Source Status": live (2xx), dead (4xx/5xx), paywall, not fetched.

Step 3: Add Action Queue entry type: "N source URLs are dead — find replacements," grouped by entity.

Fix 5: Unverifiable Subcategories (Medium-large effort, needs schema + prompt changes)

Part A — Source-fetcher error classification: Store the fetcher error type (dead_link, paywall, not_cached, topic_mismatch) as a fetchStatus field alongside the evidence. Dashboard can then show "32% unverifiable" broken down into causes.

Part B — LLM prompt refinement: When source IS fetched but doesn't confirm the claim, distinguish "source doesn't discuss this topic at all" from "source discusses it but is ambiguous." Both are currently "unverifiable."

Fix 6: Full Dashboard Restructure (Large effort, builds on 2-5)

Tab 1: Data Quality (new default)

• Big donut/bar: Confirmed / Issues / Unverifiable / Unchecked
• Accuracy rate (excluding unverifiable)
• Contradictions list — top priority action items
• Outdated list — second priority

Tab 2: Coverage (improved)

• Per-record-type progress bars with completion estimates
• "No source URL" count per type
• Source health: dead links, paywalled, not fetched

Tab 3: Action Queue (improved)

• Contradicted facts on high-traffic pages first
• Dead link replacement queue
• Source research queue (facts without sources)
• Recheck queue (stale verdicts)

Tab 4: System Health (new)

• Model freshness distribution
• Confidence histograms per verdict type
• Check velocity and cost tracking

---

Suggested Implementation Order

#	Fix	Impact	Effort	Dependencies
1	Resource linking	Medium	Small	None
2	Headline stat redesign	High	Medium	None
3	Claims display improvement	High	Medium	None
4	Broken link detection	Medium	Medium	Source-fetcher
5	Unverifiable subcategories	Medium	Medium-Large	Schema + prompt
6	Full dashboard restructure	High	Large	Builds on 2-5

Fixes 1-3 could ship together as a first PR. Fix 4-5 as a second. Fix 6 as a polish pass.

[OAGr]

Status (April 2026): Audit document with fix proposals. Source-check orchestration and recheck commands exist. The specific fixes proposed here (usability improvements, verdict accuracy, UI gaps) should be decomposed into individual issues for implementation.

[OAGr]

Decomposed into Issues

Audit reviewed and decomposed into 3 focused implementation issues covering the highest-impact fixes:

New issues from this discussion

1. Source-check dashboard: replace meaningless 'Avg Confidence' with data quality breakdown #3654 — Replace meaningless Avg Confidence with data quality breakdown (Problem 1 + Fix 2)

   • Remove Avg Confidence headline stat, replace with verified/issues/unverifiable/unchecked breakdown and accuracy rate
   • Scope: dashboard UI change in /source-checks and /internal/source-check-coverage

2. Source-check verdicts table: show actual claims instead of opaque record IDs #3658 — Show actual claims in verdicts table instead of opaque record IDs (Problem 5 + Fix 3)

   • Add claim/property column so users can see what was checked (e.g., revenue = $100M) without clicking through to detail pages
   • Scope: API + table UI change

3. Source-check: detect dead/broken source URLs before calling LLM #3663 — Detect dead/broken source URLs before calling LLM (Problem 4 + Fix 4)

   • Query citation_content.fetchStatus pre-LLM, skip LLM for dead links, make dead links distinguishable from other unverifiable causes
   • Scope: source-check pipeline + dashboard filter

Already addressed or covered by existing issues

• Problem 3 (Resource Linking): Already fixed — storeSourceCheckEvidence() in verdict-handler.ts already calls lookupResourceByUrl() when no resourceId is provided.
• Problem 2 (Unverifiable subcategories): Partially covered by Improve source-check coverage for 869 unverifiable records #3609 (reduce unverifiable rate) and Auto-suggest better source URLs when source-check returns unverifiable #3548 (auto-suggest better URLs). Source-check: detect dead/broken source URLs before calling LLM #3663 above adds the dead-link detection piece.
• Problem 6 (Full dashboard restructure): Deferred — depends on fixes 2-5 being implemented first. Source-check dashboard: replace meaningless 'Avg Confidence' with data quality breakdown #3654 covers the most impactful part (headline stats).

Implementation order

1. Source-check dashboard: replace meaningless 'Avg Confidence' with data quality breakdown #3654 (stat redesign) and Source-check verdicts table: show actual claims instead of opaque record IDs #3658 (claims display) — independent, can be done in parallel
2. Source-check: detect dead/broken source URLs before calling LLM #3663 (broken link detection) — independent but benefits from dashboard changes in Source-check dashboard: replace meaningless 'Avg Confidence' with data quality breakdown #3654

[OAGr]

Starting work on Fixes 5-6: Unverifiable subcategories in dashboard UI + tab restructure. Branch: claude/disc-3558-dashboard-restructure. Fixes 1-4 are already shipped (PR #3681 + existing code).

[OAGr]

Completed Fixes 1-5 (partially). PR #3735 adds dead link count to dashboard (Fix 5, step 1). Fixes 1-4 were already shipped in PR #3681.

Remaining for this discussion:

• Fix 5 completion: add "Source Status" filter tab (live/dead/paywall/not-fetched) + unverifiable subcategory breakdown in the table
• Fix 6: Full dashboard restructure with tabs (Data Quality / Coverage / Action Queue / System Health)

Status update: 5/6 fixes done, 1 remaining (Fix 6 — large UI work).

[OAGr]

Superseded by Discussion #3993 (Source-Check → Sourcing Rename — Implementation Plan).

The architecture and naming decisions in this discussion have been consolidated into the 7-phase sourcing rename plan. Key changes: routes become /sourcing, tables rename to record_sources/source_verdicts, components standardize on SourcingDot/SourcingBadge.

Phase 0: #4001 | Phase 1: #4002