Data Source Resources: Structured Data Verification System

[OAGr]

Problem

916 of 2,486 source-checked records (37%) are "unverifiable." The source-check system verifies structured claims against source URLs using LLM reading comprehension, but this approach fails for tabular data sources (CSVs, APIs, HTML tables).

Breakdown by record type:

Type	Unverifiable	Total	Rate	Root Cause
Investment	50	56	89%	All point to same wrong URL (SB 1047 article) — data pipeline bug
Grant	206	261	79%	Source URLs point to generic org homepages, not actual data sources
Personnel	155	411	38%	Opaque stableIds sent to LLM instead of resolved names
Division	32	91	35%	Similar patterns
Fact	464	1444	32%	Dead URLs (404s), generic homepages, source doesn't discuss specific fact

Core insight

For tabular data (grants from CSVs, personnel from databases), verification should be deterministic row-matching, not LLM reading comprehension. Parse the source CSV, find a matching row by key fields (grantee + amount + date). 100% accurate, zero cost.

Research

Investigated: DCAT (W3C), Frictionless Data/Table Schema, Schema.org Dataset, CKAN, Wikidata provenance model (stated in + record ID + retrieved date), git-scraping (Simon Willison), Datasette, RFC 7111 CSV fragment identifiers, Evidence.dev, ClaimDB benchmark (ICLR 2025), DVC, medallion architecture.

Key finding: No existing standard models provenance at the individual record level. Wikidata's three-part citation model is closest. Deterministic row-matching against tabular sources is more reliable than LLM verification (ClaimDB showed best models only reach 82.7% on large-table verification).

Architecture decisions

1. Standalone data_sources table (not a resource sub-table) — data sources identified by source ID, not URL; some have no stable URL
2. source_snapshots table — stores raw content permanently in PG (~728MB/yr at worst); content-hash dedup
3. Parse on-demand from raw_content; do NOT store parsed_rows JSONB (staleness risk, JSONB perf issues)
4. Deterministic row-matching as verification strategy for records with data source manifests; LLM as fallback

Implementation phases

Phase 0: Quick fixes (3 independent PRs)

• Fix investment source URLs (wrong URL bug)
• Resolve personnel stableIds to names before LLM verification
• Add Wayback Machine fallback for 404'd fact sources

Phase 1: Schema foundation

• data_sources table (id, format, access_method, record_type, fetch_url, column_mapping, verification_config, etc.)
• source_snapshots table (data_source_id, snapshot_hash, raw_content, fetched_at; UNIQUE dedup)
• API routes (Hono RPC) for data sources and snapshots
• Client functions in crux

Phase 2: Grant pipeline integration

• Data source manifests per grant source (schema, column mapping, verification config)
• Extend GrantSource interface with optional manifest
• Snapshot capture during grant import
• Fix grant source URLs (use actual CSV URL instead of generic org page)

Phase 3: Deterministic verification

• Source content parsers (CSV, HTML table, JSON)
• Fuzzy matching library (name normalization, edit distance, amount/date tolerance)
• Deterministic matcher (row-matching against snapshots)
• Pipeline integration (deterministic path before LLM in source-check orchestrator)

Phase 4: CLI + Dashboard

• crux tb data-sources list|show|snapshot|verify|health
• Internal dashboard (Pattern A MDX page)

Expected impact

~400-460 of 916 unverifiable records fixed (~44-50%).

Full plan

Detailed implementation plan with file paths, migration SQL, and risk analysis: ~/.claude/plans/whimsical-noodling-hippo.md

[OAGr]

Status (April 2026): The data sources system (tables, API, CLI) shipped in PRs #3571-#3572. The verification pipeline described here (structured data verification against source snapshots) extends the existing source-check system. See also #3589 for the UI integration plan.

[OAGr]

Decomposed into implementable issues

Based on the current infrastructure state (data_sources/source_snapshots tables shipped in #3571-3572, deterministic matcher working for grants, source parsers tested), here are 3 focused next-step issues:

Phase 0: Quick fixes

1. Fix investment record source URLs (89% unverifiable — all point to wrong URL) #3659 — Fix investment record source URLs (89% unverifiable — all point to wrong URL)
2. Add Wayback Machine fallback for 404'd source URLs in source-check pipeline #3664 — Add Wayback Machine fallback for 404'd source URLs in source-check pipeline

Phase 3: Expand deterministic verification

3. Extend deterministic row-matching verification beyond grants to personnel and investments #3669 — Extend deterministic row-matching verification beyond grants to personnel and investments

What already shipped

• Phase 1 schema foundation (tables + API + CLI): feat: Phase 3 — deterministic row-matching verification #3571, feat: Phase 4 — data sources CLI commands #3572
• Phase 2 grant manifests + snapshot capture: built into grant import pipeline
• Phase 3 deterministic matcher for grants: crux/lib/source-check/deterministic-matcher.ts + parsers + fuzzy matching
• Personnel stableId resolution: Source-check LLM prompt sends raw stableIds instead of human-readable names #3137 (closed)
• Source check UI integration: Data Sources UI Integration — Implementation Plan #3589

Not yet filed (future phases)

• Grant source URL fixes (79% unverifiable — generic org homepages instead of CSV URLs)
• Dashboard for data source health monitoring (Phase 4)
• Additional manifest authoring for more grant sources

[OAGr]

Superseded by Discussion #3993 (Source-Check → Sourcing Rename — Implementation Plan).

The architecture and naming decisions in this discussion have been consolidated into the 7-phase sourcing rename plan. Key changes: routes become /sourcing, tables rename to record_sources/source_verdicts, components standardize on SourcingDot/SourcingBadge.

Phase 0: #4001 | Phase 1: #4002

[OAGr]

Branch: claude/deploy-tasks-verify-cmd

Closing as superseded. The data sources / structured verification work is now consolidated under #3993 (Rename source-check → sourcing) as part of the broader cleanup. See also #4017 (Data integrity epic). — Discussion review 2026-04-08.