Deterministic Wikidata Source-Check Matcher

[OAGr]

Deterministic Wikidata Source-Check Matcher — Implementation Plan

TL;DR

~425 FactBase facts sourced from Wikidata are verified by an LLM reading noisy HTML — expensive ($4.25/week) and less reliable than a direct API comparison. This plan adds a deterministic Wikidata matcher: extract Q-numbers from source URLs, batch-fetch structured claims via wbgetentities, and compare property values directly. Phase 1 covers 12 property types (~414 facts, 97%). Numeric properties (revenue, headcount — 11 facts) are deferred to a follow-up. Top open question: whether lazy label resolution is fast enough or if batch pre-fetch for labels is needed.

Problem

The weekly source-check spends ~$4.25 and ~425 LLM calls verifying Wikidata-sourced facts by converting HTML to text and asking Claude Haiku to judge. Wikidata has a structured JSON API that returns exact property values. The deterministic matcher for grants (PR #3571) proved the "deterministic first, LLM fallback" architecture works — this extends it to Wikidata facts.

Current State

Asset	Status	Gap
crux/lib/source-check/item-verifier.ts	Has tryDeterministicMatch() for grants (line 227)	No Wikidata matching — facts fall through to LLM
crux/lib/source-check/fuzzy-match.ts	nameMatches(), amountMatches(), dateMatches()	Missing urlMatches() for website/wikipedia-url
crux/commands/wikidata-enrich.ts	WIKIDATA_PROPERTY_MAP (10 PIDs, lines 60-154)	Enrichment direction only; verification needs reverse map
crux/commands/orgs.ts	getWikidataClaims(), getWikidataLabel(), fetchWithRetry()	Duplicated in people/enrich.ts — shared API code extracted into matcher
packages/factbase/data/things/*.yaml	425 facts, 138 unique QIDs, 137 entities	QIDs only in source URLs; ~136 facts have trailing commas in URLs
crux/lib/source-check/source-check-recheck.ts	Recheck always uses LLM	Would cause verdict flip-flopping without Wikidata integration

Proposed Approach

Add tryWikidataMatch() in item-verifier.ts that intercepts fact items with Wikidata source URLs. Uses wbgetentities API (batched 50 QIDs/call) with lazy memo cache for label resolution. Falls through to LLM on any failure.

Per-fact verification (tryWikidataMatch):
  1. Extract QID from source URL via regex /Q(\d+)/
  2. Fetch entity claims from memo cache (or wbgetentities API)
  3. Map FactBase property → Wikidata PID
  4. Filter claims: preferred rank first, then normal. Never deprecated.
  5. Extract + normalize Wikidata value
     - Entity-refs (Q84): resolve label via memo cache (or wbgetentities)
     - Sitelinks (wikipedia-url): construct URL from title
  6. Compare against fact value using typed comparison
  7. Return VerifyResult or null (→ LLM fallback)

Batch path optimization: In orchestrator.ts, scan items before verification loop → collect unique QIDs → batch pre-fetch entities (3 calls for 138 QIDs). Individual facts then hit the memo cache with zero API calls.

Realtime path (--entity=X): Lazy per-fact fetching with memo cache. First fact for a QID triggers one API call; subsequent facts for the same QID are free. Typically 1-2 API calls per entity.

Key Decisions

• wbgetentities not SPARQL: Simpler, supports batching (50 QIDs/call = 3 calls for all entities), no SPARQL endpoint queue. SPARQL stays in enrichment commands.
• Lazy memo cache for labels, not orchestrator pre-fetch: Lazy approach avoids orchestrator coupling. Each label resolution is one wbgetentities call, cached for reuse. Worst case for full corpus: ~80 individual label calls vs 6 batched. Acceptable for a weekly job.
• New wikidata-matcher.ts, not extending deterministic-matcher.ts: Different data shape (structured claims vs tabular rows). Clean module boundary.
• 12 properties in v1, numeric deferred: revenue and headcount need Wikidata quantity parsing ("+500" with unit URIs, currency handling). 11 facts. Deferred to follow-up.
• No circuit breaker: try/catch returning null is sufficient for a weekly batch job. LLM fallback is the circuit breaker.
• No separate shared API module: Single consumer (the matcher). Extract when a second consumer appears.

Architecture

File	Action	Purpose
crux/lib/source-check/wikidata-matcher.ts	create (~250 lines)	QID extraction, property-to-PID map (12 properties), wbgetentities with memo cache, entity-ref label resolution, sitelink URL construction, typed comparison, claim rank filtering, tryWikidataMatch() entry point
crux/lib/source-check/wikidata-matcher.test.ts	create (~200 lines)	Test each property type with mock API fixtures. Edge cases: multi-valued claims, missing properties, deprecated claims, trailing-comma URLs, sitelink title with special chars, non-English label fallback
crux/lib/source-check/item-verifier.ts	modify (~15 lines)	Add tryWikidataMatch() call for kind === 'fact' items with Wikidata URLs, before LLM path
crux/lib/source-check/orchestrator.ts	modify (~20 lines)	Batch path: scan items for QIDs, pre-fetch entities via batched wbgetentities before verification loop
crux/lib/source-check/fuzzy-match.ts	modify (~15 lines)	Add urlMatches(): strip trailing /, normalize http↔https, strip www
crux/lib/source-check/fuzzy-match.test.ts	modify (~20 lines)	Tests for urlMatches()
crux/commands/source-check-recheck.ts	modify (~10 lines)	Add tryWikidataMatch() before LLM call to prevent verdict flip-flopping

Property-to-PID verification map (12 properties, ~414 facts):

FactBase Property	Wikidata PID	Facts	Comparison Strategy
website	P856	97	urlMatches() — normalize trailing /, http/https, www
wikipedia-url	sitelinks.enwiki	74	Construct URL from title, then urlMatches()
founded-date	P571	46	dateMatches() — hierarchical year/month/day
country	P17	42	Entity-ref → label → nameMatches()
headquarters	P159	39	Entity-ref → label → nameMatches()
founded-by-name	P112	30	Entity-ref → label → nameMatches(), multi-valued OR
education	P69	22	Entity-ref → label → nameMatches(), multi-valued OR
legal-structure	P1454	19	Entity-ref → label → nameMatches()
born-year	P569	18	Year extraction → dateMatches()
ceo	P169	5	Entity-ref → label → nameMatches(), prefer rank
parent-organization	P749	5	Entity-ref → label → nameMatches()
description	N/A	2	Skip — no PID. Falls through to LLM.
revenue	P2139	7	Deferred — needs quantity/currency parsing
headcount	P1128	4	Deferred — needs quantity parsing

Verdict mapping:

• Exact/fuzzy match → confirmed (confidence 0.95)
• Value differs → contradicted (confidence 0.90)
• Property missing in Wikidata → unverifiable (confidence 0.80)
• No PID mapping / API failure → return null (fall through to LLM)
• Partial match (e.g., one of two founders found) → partial (confidence 0.70)

Evidence format:

• extractedValue: "Wikidata P159 (headquarters) = Q62 (San Francisco)" — raw QID + resolved label
• reasoning: "[wikidata-api] FactBase value 'San Francisco' matches Wikidata P159 label 'San Francisco' (fuzzy name match). Claim rank: preferred."
• checkerModel: 'wikidata-api'

Implementation Phase: Wikidata Matcher — M (1 session)

Goal: All 414 Wikidata-sourced facts (12 properties) verified deterministically via API. Zero LLM cost for these facts.

• Create wikidata-matcher.ts — QID extraction (regex /Q(\d+)/ handles trailing commas), property map, memo cache for entities + labels, typed comparison per property, claim rank filtering (preferred > normal, never deprecated), sitelink URL construction, multi-valued OR for founder/education
• Create wikidata-matcher.test.ts — all 12 property types with mock fixtures, edge cases
• Add urlMatches() to fuzzy-match.ts + tests
• Modify item-verifier.ts — tryWikidataMatch() call for Wikidata-sourced facts
• Modify orchestrator.ts — batch QID pre-fetch
• Modify source-check-recheck.ts — add tryWikidataMatch() before LLM
• Smoke test: WIKI_SERVER_ENV=prod pnpm crux fb source-check --entity=80000-hours --verbose
• Smoke test: WIKI_SERVER_ENV=prod pnpm crux fb source-check --entity=deepmind --verbose
• Verify: all Wikidata facts show checkerModel: 'wikidata-api'
• Compare verdict distribution against last LLM run (expect similar confirmed/partial/contradicted counts)

Quality gates:

• pnpm test passes (all existing + new tests)
• pnpm build succeeds
• Smoke tests produce same verdicts as previous LLM run (5/6 confirmed, 1/6 partial for 80000-hours)
• Full dry-run: pnpm crux fb source-check --dry-run shows correct item count

Exit criteria: WIKI_SERVER_ENV=prod pnpm crux fb source-check --entity=80000-hours --verbose shows all facts with checkerModel: 'wikidata-api'.

Rollout sequence: Per-entity smoke tests → compare verdict distribution → full corpus run. If >20% contradicted rate (vs ~5% from LLM runs), investigate before proceeding.

Scope Cuts

• Numeric properties (revenue, headcount): 11 facts. Need Wikidata quantity parsing ("+500" with unit URIs, currency comparison). Follow-up issue.
• Enrichment consolidation: Triplicated API code in orgs.ts/people/enrich.ts/wikidata-enrich.ts works fine. Consolidate when a second consumer of shared API client emerges. Follow-up issue.
• Stored wikidata-qid property: QIDs extracted from source URLs at verification time. Persistent QID field is a separate enhancement.
• Auto-correction of contradicted facts: Matcher stores verdicts but doesn't auto-fix. Improve pipeline handles corrections.
• Dashboard UI changes: Existing /source-checks page displays all checker models. Only change: add 'wikidata-api' case to formatCheckerModel() (~1 line).

Quality & Verification Infrastructure

Tests: wikidata-matcher.test.ts (~200 lines). Mock Wikidata API at HTTP level. Test each of 12 property types. Edge cases: multi-valued P112 (4 founders), missing P169, deprecated claim rank, trailing comma URLs, sitelink title with parentheses ("Asana (software)"), entity with no English label (fallback to null → LLM), multi-QID entity (facts from different QIDs on same FactBase entity).

Integration smoke test:

WIKI_SERVER_ENV=prod pnpm crux fb source-check --entity=80000-hours --verbose
WIKI_SERVER_ENV=prod pnpm crux fb source-check --entity=deepmind --verbose

UI verification: N/A — no UI changes beyond 1-line formatCheckerModel addition.

Deploy steps: No new env vars, no migrations. Standard PR merge. Wikidata API is public (no auth).

Documentation: N/A — the matcher follows the existing tryDeterministicMatch pattern. Property map in wikidata-matcher.ts is self-documenting.

Monitoring: After first weekly run, compare verdict distribution (confirmed/contradicted/partial counts) against previous LLM-based run. If contradicted count increases by >2x, file an issue.

Risks & Mitigations

Risk	Severity	Mitigation
Entity-ref label resolution (8/12 properties)	Significant	Lazy memo cache. nameMatches() handles normalization. Fallback: no English label → return null → LLM.
Multi-valued properties (founder, education)	Significant	OR semantics: match any Wikidata value. partial if some found, contradicted only if none found.
Trailing commas in ~136 source URLs	Minor	Regex /Q(\d+)/ extraction (not URL parsing). Test with exact format.
Deprecated Wikidata claims (stale CEO, old HQ)	Significant	Filter: preferred rank first, then normal. Never deprecated. Test with fixture containing deprecated + preferred claims for same PID.
Wikipedia URL from sitelinks (title, not URL)	Significant	Construct URL: https://en.wikipedia.org/wiki/${title.replace(/ /g, '_')}. Test with parenthesized titles. Fetch with props=claims|sitelinks.
Verdict flip-flopping on rechecks	Significant	Integrate tryWikidataMatch() into recheck path. Same matcher runs on rechecks.
Systematic bug overwrites 414 aggregate verdicts	Significant	Rollout: per-entity first, compare distributions, then full run. Old LLM evidence preserved in evidence table (different checkerModel in dedup key).
API downtime during weekly run	Minor	try/catch returns null → LLM fallback. No worse than today.

Open Questions

• Lazy vs batch label resolution: Lazy is simpler (no orchestrator coupling) but makes more individual API calls (~80 vs ~6 batched). For a weekly job, is the difference (~60 seconds) worth the complexity? Current plan: lazy.
• Rate limit setting: Existing enrichment uses 1s/request. Wikidata allows ~200 req/min (300ms). Should the matcher use the faster rate? Current plan: match existing 1s setting for safety.

Rejected Approaches

• A: Minimal (skip entity-ref properties): 76% coverage. The effort difference is small (~50 lines for label resolution).
• C1: Enrichment-as-Verification: Couples verification to enrichment correctness. Uses SPARQL (heavier).
• C2: Snapshot and Match: Adds infrastructure (cache table, refresh job) for a problem batched wbgetentities solves in 3 calls.
• C3: Wikidata Watches Us: Needs nonexistent import timestamps. First run can't shortcircuit.
• D: 4-phase incremental: More total effort for same outcome.

Red Team Log

Technical critic — 2 blocking, 10 significant:

• [FIXED] Blocking: realtime execution path not addressed → added tryWikidataMatch in verifySingleItem
• [FIXED] Blocking: recheck system causes verdict flip-flopping → integrated matcher into recheck path
• [FIXED] Claim rank filtering (preferred > normal, never deprecated)
• [FIXED] Wikipedia URL from sitelinks needs URL construction from title
• [FIXED] Trailing commas affect ~136 facts, not ~25
• [FIXED] Revenue/headcount temporal qualification → deferred numeric properties entirely
• [FIXED] Entity-ref with no English label → fallback to null
• [ACCEPTED] Pre-fetch breaks item-at-a-time abstraction → lazy cache for labels, batch only for entities
• [ACCEPTED] ~12 API calls at 1s/req = ~15s wall time (acceptable for weekly job)

Scope critic — key cuts adopted:

• [ADOPTED] Dropped Phase 2 (enrichment consolidation) → follow-up issue
• [ADOPTED] Dropped separate wikidata-api.ts → code in matcher
• [ADOPTED] Dropped circuit breaker → try/catch sufficient
• [PARTIALLY ADOPTED] Ship 12 properties (not 8) since born-year/ceo/parent-org are trivial. Only deferred numeric (revenue/headcount).
• [ADOPTED] Lazy label resolution instead of orchestrator pre-fetch for labels

Integration critic — 1 blocking, 4 significant:

• [FIXED] Blocking: recheck integration to prevent flip-flopping
• [FIXED] Evidence format specified (extractedValue, reasoning, checkerModel)
• [FIXED] Rollout sequence: per-entity → compare → full corpus
• [FIXED] Aggregate verdict overwrite acknowledged; old LLM evidence preserved in evidence table
• [ACCEPTED] Shadow comparison: document expected distribution, compare after first run (manual check, not automated)