Verified-by-Default Data Enrichment — Incremental Rollout Plan

[OAGr]

Verified-by-Default Data Enrichment — Incremental Rollout Plan

TL;DR

94% of TableBase records have zero source-check coverage despite the verification infrastructure being fully built (source-check orchestrator, claims pipeline, inline verification, pre-submit gates, verification dots UI — all shipped, mostly unused). The source-check orchestrator handles all 13 record types today but has a routing bug that prevented type-specific runs. This plan fixes that bug, then systematically activates existing verification across record types, starting with personnel and expanding outward. Core code work is ~8-10 sessions; the remaining weeks are operational runs (backfills, triage, URL fixes). Phase 1 starts with a 1-line bug fix.

Problem

Data enters the system unverified. The verification infrastructure exists but agents bypass it — verification is optional and adds latency. Result: ~250 personnel records with 0 verdicts, ~3,000 grants mostly unchecked, and no enforcement preventing more unverified data from entering.

The fix isn't more infrastructure — it's activating what exists, backfilling coverage, and gradually tightening enforcement.

Current State

Asset	Status	Gap
Source-check orchestrator (crux tb verify)	Working, all 13 record types	Routing bug: tb verify personnel doesn't pass table param → verifies ALL types
Claims pipeline (proposed_claims, claim-verification worker)	Fully built (#3253), 0 usage	Agents skip it; no enforcement
Pre-submit verification (pre-submit-verification.ts)	Active, default ON	Only catches contradictions at submit time
Inline verification (write-inline-verdicts.ts)	Built, never invoked	0 records have inline verification
Resource suggest API (/api/resources/suggest)	Working	Not integrated into enrichment workflow
Deterministic row-matching	Grants only	Not extended to investments, personnel
Verification dots UI	Shipped, all surfaces	Low coverage → mostly empty dots
Recheck scheduling (next_check_due field)	Field exists	Nothing reads it (#3661)
Personnel records (PG)	~250	0 verification verdicts
Grants (PG)	~3,000	Mostly unchecked
Person entities (YAML)	749	87% lightweight, 7% have sources
Resources	5,346	98% summaries, 9% authors/dates

Key Decisions

• Decision: Orchestrator is the primary verification path for both backfill and ongoing verification

  • The orchestrator works on all 13 record types with near-zero code changes. It verifies records against their source URLs — the same thing the claims pipeline does, but without requiring a separate job worker, polling loop, or table.
  • The claims pipeline (Claims-first verification architecture for TableBase data #3253) is a stretch goal for audit-trail-on-new-writes, not a prerequisite for verification coverage.
  • Rationale: The claims pipeline was "fully built, 0 usage" — a signal that the complexity doesn't fit the workflow. The orchestrator is proven and handles the same use case.

• Decision: Backfill first, enforce gradually

  • Phase 1-2: Backfill coverage (no enforcement)
  • Phase 3: Advisory warnings when records lack verification
  • Phase 4: Soft block (--force override required)
  • Phase 5: Hard block for types with >70% coverage
  • Rationale: Hard-blocking at 0% coverage means agents can't submit anything.

• Decision: Focus on 5 high-value record types

  • Priority: personnel → grants → investments → funding rounds → FactBase facts
  • Low-volume types (assessments, equity positions, secondary market prices): run ad-hoc
  • Rationale: These 5 cover ~90% of user-visible records.

Architecture

File	Action	Phase	Purpose
crux/commands/source-check-orchestrate.ts	Fix (1 line)	1 (wk 1)	Pass table: mapped to orchestrator options
crux/commands/source-check-orchestrate.test.ts	Create/modify	1 (wk 1)	Test: tb verify personnel only collects personnel records
crux/lib/source-check/item-collectors.ts	Modify (1 line)	1 (wk 2)	Change && to || in stableId skip guard (line ~304)
crux/lib/source-check/item-collectors.test.ts	Create/modify	1 (wk 2)	Test: resolvable org + unresolvable person → skip
crux/lib/source-check/orchestrator.ts	Modify	2 (wk 10)	Add batch chunking for 3,000+ record sets
crux/lib/source-check/deterministic-matcher.ts	Modify	3 (wk 13)	Extend from grants-only to investments
crux/lib/source-check/record-fields.ts	Modify	3 (wk 13)	Add stableId skip guards for funding-round, benchmark-result
.claude/commands/review-tablebase-enrich.md	Modify	2 (wk 9)	Add "run tb verify after submit" step to enrichment skill
crux/validate/validate-verification-coverage.ts	Modify	4 (wk 18)	Soft block: require --force to bypass

No new PG tables. One small migration only if claims stretch goal is pursued (otherwise none).

Implementation Phases

Phase 1: Personnel Verification (Weeks 1-6) — 2-3 code sessions + ops

Goal: Personnel records show verification dots. Unverifiable rate drops below 15%.

• Week 1 [code]: Fix routing bug in source-check-orchestrate.ts — add table: mapped to options (line ~115). Add test. Run WIKI_SERVER_ENV=prod pnpm crux tb verify personnel --dry-run to assess baseline.
• Week 1 [ops]: Run pnpm crux tb verify personnel --batch (~$1.25). Report verdict distribution.
• Week 2 [code]: Tighten name-resolution guard: change && to || at item-collectors.ts:304 so records with unresolvable person names are skipped even if org name resolves. Add test.
• Week 2 [ops]: Re-run on previously-skipped records.
• Week 3 [ops]: Audit "unverifiable" results. Categorize: dead URL, wrong page, JS-rendered, paywall.
• Weeks 4-5 [ops]: Fix source URLs using suggestResources(). Register top-50 org team pages in website_source_pages. Re-run source-check on updated records.
• Week 6 [ops]: Report: personnel verdict distribution before/after. Decide dot visibility threshold (Open Question Basic Search #2).

Quality gates: tb verify stats --table=personnel shows verdicts. Dots visible on 3+ org pages.
Exit criteria: >60% of personnel have verdicts. Unverifiable <15%.

Phase 2: Grants + Workflow Integration (Weeks 7-14) — 3-4 code sessions + ops

Goal: Grants verified. Enrichment skill runs verification after each submission.

• Week 7 [ops]: Run pnpm crux tb verify grant --dry-run. Assess: how many have source URLs? What's the deterministic-match rate?
• Week 8 [ops]: Run grants backfill in test batch (100 records). Verify deterministic matcher works on tabular grant sources.
• Week 9 [code]: Update enrichment skill (.claude/commands/review-tablebase-enrich.md) to add post-submit verification step: after crux tb submit, run crux tb verify --type=record --table=<type> --entity=<id>. This gives immediate feedback on what was just submitted.
• Week 10 [code]: Add batch chunking to orchestrator for 3,000+ record sets (split into pages of 500, sequential API calls). Run full grants backfill (~$5-10).
• Weeks 11-12 [ops]: Triage contradicted grants. Fix source URLs. Re-run.
• Week 13 [code]: Extend deterministic matcher to investments. Add stableId skip guards for funding-round, benchmark-result in record-fields.ts.
• Week 14 [ops]: Run investments backfill (~1,000 records). Run funding-rounds backfill (~500).

Quality gates: Grants >80% coverage. Deterministic matcher handles investments. Enrichment skill includes post-submit verification.
Exit criteria: Financial record types show dots on directory pages. <5% contradicted.

Phase 3: Remaining Types + Advisory Enforcement (Weeks 15-22) — 2-3 code sessions + ops

Goal: All priority types verified. Advisory warnings when records lack verification.

• Weeks 15-16 [ops]: Publications backfill (DOI resolution for deterministic checks where possible). FactBase facts backfill (crux tb verify --type=fact).
• Weeks 17-18 [code]: Add advisory enforcement — enrichment skill logs a warning when records are submitted without running tb verify afterward. The gate check (validate-verification-coverage.ts) starts printing warnings for manifests with zero verification.
• Weeks 19-20 [ops]: Benchmark results, entity events, remaining types — run orchestrator ad-hoc.
• Weeks 21-22 [ops]: Global audit: coverage across all types. Spot-check 10 "confirmed" verdicts per type for false-positive rate.

Quality gates: All 5 priority types >50% coverage. False-positive rate <10%.
Exit criteria: E2200 dashboard shows coverage across all major types. Advisory warnings active.

Phase 4: Soft Enforcement (Weeks 23-30) — 1-2 code sessions + ops

Goal: Submitting without verification requires explicit opt-out.

• Week 23 [code]: Soft enforcement in validate-verification-coverage.ts — fail gate check for manifests with zero verification unless --force is used. Log all --force usages.
• Weeks 24-26 [ops]: Monitor: do any agent sessions hit the soft block? Fix workflows that don't include verification.
• Weeks 27-28 [ops]: First quarterly recheck — run orchestrator on all types, targeting stale verdicts (>90 days old). Fix any verdict flips.
• Weeks 29-30 [ops]: Personnel enrichment sessions for under-covered orgs (using updated skill with post-submit verification).

Quality gates: Soft enforcement active. <5 --force overrides per week.
Exit criteria: All priority types >70% coverage. Enrichment sessions routinely include verification.

Phase 5: Hard Enforcement + Steady State (Weeks 31-50) — background ops

Goal: Verification is structural norm. System self-maintains.

• Week 31 [code]: Hard enforcement for personnel (>70% coverage). requireVerification=true on personnel sync endpoint for agent sessions.
• Weeks 32-35 [ops]: Expand hard enforcement to each type as it reaches >70%. Continue enrichment.
• Weeks 36-40 [ops]: Second quarterly recheck. Assess whether automated scheduling is justified by flip rate.
• Weeks 41-45 [ops]: If flip rate >5%, implement simple recheck cron. Otherwise, continue quarterly manual runs.
• Weeks 46-50 [ops]: Annual retrospective: total coverage, cost, false-positive rate, flip rate. Evaluate stretch goals.

Quality gates: Hard enforcement for personnel. Quarterly rechecks running.
Exit criteria: 0 unverified personnel records submitted in last 30 days.

Stretch Goals (evaluated after Phase 3)

These are valuable but not essential for the core verification pipeline:

1. Claims pipeline integration: Wire proposed_claims into enrichment as an audit trail for new writes. Requires solving the sequencing issue (claims need to reference records that don't exist yet — either use placeholder IDs or verify claims before record creation). Evaluate at Week 15 based on whether the audit trail gap is actually a problem.

2. crux research <entity> command: Unified discover → resources → verify → write pipeline. Evaluate at Week 20 based on whether the enrichment skill update (Phase 2, Week 9) is sufficient.

3. Website-as-data-feed extraction (LLM fact extraction from website snapshots into FactBase #3655): LLM extraction from org team page snapshots. Separate feature — evaluate at Week 50 retrospective.

4. Automated recheck scheduling (Add automated recheck scheduling via GitHub Actions #3661): Cron-based recheck using next_check_due field. Evaluate at Week 40 based on quarterly recheck flip rate.

Scope Cuts

• No new PG tables or migrations for core plan (claims stretch goal would need one).
• No Playwright for JS-rendered pages (Use Playwright JS rendering for source-check on JS-heavy pages #3546) — accept "unverifiable"; fix via better source URL selection.
• No PR manifests for TableBase changes — verdicts table is the audit trail.
• No cross-page consistency checking — separate problem.
• No entity-level aggregate badges (Entity-level aggregate verification badge for org/person headers #3445) — deferred until coverage >80%.
• Low-volume types are ad-hoc — no dedicated phases.

Quality & Verification Infrastructure

Area	Plan
Tests	Wk 1: routing fix test. Wk 2: name-resolution guard test. Wk 10: batch chunking test. Wk 13: deterministic matcher extension test.
Migration verification	Wk 13: grep for recordType === 'grant' guards and expand all.
Integration smoke test	Each backfill: tb verify stats before/after. tb verify <type> --dry-run --limit=5 spot-check.
UI verification	After each backfill: load 3 org pages, check dots. Check E2200 dashboard.
Deploy steps	Wk 1: normal PR for routing fix. No new env vars. Verify wiki-server is running for WIKI_SERVER_ENV=prod commands.
Documentation	Wk 9: enrichment skill updated. Wk 23: document enforcement policy in CLAUDE.md.
Monitoring	E2200 dashboard. Post-backfill: check confidence distribution (>90% confirmed = suspicious).

Risks & Mitigations

Risk	Severity	Mitigation
tb verify personnel routing bug	Blocking	Week 1: 1-line fix + test
Personnel name resolution — bare stableIds	Significant	Week 2: && to || guard change + test
Source URL quality — 38% unverifiable	Significant	Weeks 4-5: fix URLs before dots become meaningful
Batch chunking missing for 3,000+ grants	Significant	Week 10: add chunking before grants backfill
False-positive verdicts at scale	Significant	Spot-check 10 verdicts per type after each backfill
Enforcement cliff	Minor	Gradual: advisory (wk 18) → soft (wk 23) → hard (wk 31+)
Cost at scale	Minor	~$20 total first pass. Batch API = 50% discount.

Open Questions

1. Dot visibility threshold: Should dots be hidden for a record type until >50% of that org's records are checked? Showing 5 green dots out of 27 may mislead users. Decide after Phase 1 data.

2. Unverifiable records under enforcement: When hard enforcement activates, should "unverifiable" records be allowed? Many legitimate records have PDF/LinkedIn sources that can't be machine-read. Proposed: allow if source URL is a valid Resource record.

3. Recheck frequency: Quarterly manual sufficient, or does flip rate justify automation? Decide at Week 40 with data.

4. Claims pipeline value: Is the orchestrator-only path sufficient, or does the audit trail from claims justify the integration complexity? Decide at Week 15 with operational experience.

Rejected Approaches

• Full crux research command first: 5-7 sessions before any visible value. Backfill delivers verdicts in Week 1.
• Claims pipeline as core (not stretch): Adds complexity for marginal benefit. The orchestrator handles backfill AND post-write verification. Claims add audit trail but the sequencing issue (claims before records exist) is unsolved.
• Enforcement first: Blocks agent enrichment at 0% coverage.
• All 13 types in dedicated phases: Diminishing returns. 5 priority types + ad-hoc for rest.
• Website extraction bundled in: Separate feature (LLM fact extraction from website snapshots into FactBase #3655).
• 10-phase / 50-week active plan: Over-engineered. Core code work is ~8-10 sessions; remaining weeks are ops.

Related Issues & Discussions

#	Title	Relationship
#3209	Verified-by-Default Data Pipeline	Parent architectural vision — this plan is a pragmatic subset
#3119	Source-Check Expansion	This plan executes Phases 3-5 of that discussion
#3580	Bulk Resource Discovery	Resource enrichment feeds better source URLs into this pipeline
#3741	Source-Check Coverage Strategy	This plan answers the strategy questions raised there
#3567	Data Source Resources	Deterministic matching extension draws from this work
#2928	Websites as Data Feeds	Stretch goal — team page extraction
#2918	Enrich people entity data	Phase 1 directly addresses this
#3536	End-to-end claims pipeline test	Stretch goal prerequisite
#3661	Automated recheck scheduling	Phase 5 stretch goal
#3667	Source-check backfill across all types	This plan is the execution plan for that issue
#3669	Deterministic matching for personnel/investments	Phase 3

Red Team Log

Round 1 — Technical Critic

Blocking (fixed): tb verify personnel routing bug — command verifies ALL types, not just personnel. The mapped value is computed but never passed as table in options. Fixed: moved to Week 1, 1-line fix.

Significant (fixed): Claims pipeline requires running job worker — added deployment audit to stretch goals prerequisite.

Significant (fixed): Enforcement at Week 47 too late — moved to gradual ramp starting Week 18.

Significant (addressed): Two verification paths with no dedup — simplified to orchestrator-only as primary path. Claims moved to stretch goal. Dedup is moot if only one path is active.

Significant (noted): No orchestrator integration test — routing fix test added to Week 1.

Minor (fixed): isResolvableName guard description clarified — && to ||.

Round 1 — UX Critic

High (addressed): 38% unverifiable orange dots make site look worse — added Open Question #1 (dot visibility threshold), URL remediation before dots become meaningful.

High (fixed): 37-week gap before single command — moved crux research to stretch goal. Enrichment skill update at Week 9 covers the workflow with existing commands.

Medium (fixed): Enforcement cliff — 3-stage ramp (advisory → soft → hard).

Round 1 — Scope Critic

Applied: Compressed 10 phases → 5 phases. Cut website extraction, automated recheck, crux research command to stretch goals. Focus on 5 types not 13.

Applied: Made claims pipeline a stretch goal, not core. Orchestrator alone covers the use case.

Debated: Scope critic suggested 14 weeks total. Plan retains 50-week timeline because: (a) user explicitly asked for 50-week trajectory, (b) core code is ~10 sessions but operational runs (backfills, triage, URL fixes) take real elapsed time, (c) Phase 5 is genuine steady-state maintenance.

Round 2 — Revision Agent

Addressed: verification_source column dropped — avoids unnecessary migration. Existing checker_model distinguishes paths if needed.

Addressed: submit-verified sequencing issue (claims need target_id) — moved claims integration to stretch goal rather than trying to solve the design problem now.

Addressed: Enforcement mechanism checks manifests vs tb submit — clarified that enforcement applies to both gate check (manifests) and enrichment skill (workflow).

Noted: Reframing agent argued plan could be 6 weeks if claims are dropped entirely. This is true for the code work, but operational execution (running backfills across types, fixing URLs, triaging results) takes real calendar time. The 50-week plan reflects a background project cadence (1-2 hours/week), not padding.

Round 2 — Implementability Agent

Confirmed: Phases 1-2 are concrete and implementable. Phase 1 (Weeks 1-2 code parts) could be one session.

Addressed: Weeks marked as [code] vs [ops] for clarity.

Noted: Phases 4-5 are vague because they're operational, not engineering. This is appropriate — they don't need code specs, they need execution.

Fixed: crux/tablebase/ directory doesn't exist — submit-verified moved to stretch goals, directory issue moot.

[OAGr]

Branch: claude/deploy-tasks-verify-cmd

Closing as superseded by #3958 (Source-Check Quality Push), which explicitly pivots away from the claims-pipeline approach proposed here. The same verified-by-default vision is preserved in #3209 as the vision anchor. — Discussion review 2026-04-08.