Verified-by-Default Data Pipeline: Requiring Verification at Write Time

[OAGr]

Context & Motivation

Today we have a mature verification infrastructure — unified source_check_evidence + source_check_verdicts tables, LLM-powered source-checking via crux source-check orchestrate, semantic diff for wiki pages, citation accuracy tracking — but it all runs post-hoc. Data goes into the system first, verification happens later (if at all). The result:

• 94% of TableBase records have zero source-check coverage (Run source checks on all remaining TableBase record types #3161)
• FactBase facts can be written with inaccurate numbers — we recently found Apart Research's sprint count was listed as "55+ sprints with 6,000+ participants" when the actual source says 42 sprints with 3,000 participants
• Content pipeline creates pages without footnote citations (Content pipeline creates pages without footnote citations #3005) — LLMs hallucinate claims that contradict our own verified FactBase data
• Personnel records go straight to PG with no review — agents run crux tb tablebase submit and records appear in production immediately
• No audit trail for PG-primary data — YAML entities and FactBase facts have git history; personnel, grants, funding rounds, investments, divisions, and benchmark results have only DB timestamps

The core problem: verification is an afterthought, not a prerequisite. We need data and verification to be submitted together, atomically, as a single operation.

What triggered this

During a recent Apart Research enrichment session, an agent:

1. Dispatched a research agent to gather facts
2. Dispatched a separate verification agent to check those facts
3. Found 4 factual errors (wrong participant counts, wrong job titles, wrong sprint counts)
4. Manually corrected the data
5. Submitted personnel records to PG with no verification attached
6. Wrote a wiki page that the content pipeline populated with hallucinated claims (calling someone "Co-Director" when our own verified data said "Board Member")

This is a 10-step manual workflow where verification is step 2 of 10 and easily skipped. We need it to be step 0.

Problem Statement

Three disconnected loops:

Loop 1 (Creation):    Agent researches → writes data → commits
Loop 2 (Verification): Cron/human runs source-check → finds errors → files issues  
Loop 3 (Correction):  Agent picks up issue → fixes data → commits

Should be one loop:

Research → Extract claims → Verify each claim → Write ONLY verified claims → Upload data + verdicts together

Current State: What Exists

System	Status	Gap
Source-check pipeline (crux source-check orchestrate)	Active, unified tables	Runs post-hoc, not pre-write
Semantic diff (crux/lib/semantic-diff/)	Active on improve --apply	Warns but never blocks; not wired into enrichment
Citation accuracy (citation_quotes table)	Active	Only checks existing footnotes, doesn't find missing ones
Recheck scheduling (nextCheckDue field)	Field exists in verdicts table	Nothing reads it (#3021)
Content verification tiers (E905)	Design doc only	Not implemented
Website-as-data-feeds (Discussion #2928)	Fully designed	Not implemented
Uncited claim verification (Discussion #3087)	Surveyed external tools	Not built
Research playbooks (#3004)	Issue filed	Not built

Related discussions: #2950 (unified verification architecture), #2928 (websites as data feeds), #3087 (external research patterns), #3119 (source-check expansion).

The TableBase Change Tracking Problem

This is the most urgent gap. TableBase data is PG-primary, so changes bypass git entirely:

Data Layer	Change tracked in	Review process	Verification
Entity YAML	Git (PR)	PR review, gate checks	Gate can enforce
FactBase YAML	Git (PR)	PR review, gate checks	Gate can enforce
Wiki pages (MDX)	Git (PR)	PR review, semantic diff	Semantic diff warns
Personnel	PG only	None	None
Grants	PG only	None	None
Funding rounds	PG only	None	None
Investments	PG only	None	None
Divisions	PG only	None	None
Benchmark results	PG only	None	None

When an agent submits personnel records, they go straight to production PG. No PR, no diff, no review, no verification. The only audit trail is createdAt/updatedAt timestamps.

Proposed Architecture: Verified-by-Default Data Pipeline

Design Principle

No data enters the system without a verification verdict. The verdict may be "verified", "unverifiable", or "contradicted" — but it must exist. Unverified data is rejected at the API level.

Component 1: API-Level Verification Enforcement

Modify all wiki-server submit endpoints to require verification alongside data:

// POST /api/personnel/sync — new schema
{
  "items": [
    {
      "personId": "new: Jaime Raldua Veuthey",
      "organizationId": "5WqgebIWFQ",
      "role": "CEO",
      "roleType": "key-person",
      "source": "https://apartresearch.com/impact",
      "verification": {
        "verdict": "verified",
        "evidence": "Impact page lists 'Jaime Raldua Veuthey -- CEO' under team section",
        "sourceContentHash": "sha256:abc123...",
        "checkedAt": "2026-03-25T10:00:00Z",
        "checkedBy": "claude-haiku-4-5"
      }
    }
  ]
}

The API:

1. Rejects records without verification (unless --skip-verification flag, which is logged as an override and visible on dashboards)
2. Writes the record to its table
3. Writes the verdict to source_check_verdicts in the same transaction
4. Returns both record IDs and verdict IDs

This applies to all PG-primary tables: personnel, grants, funding rounds, investments, divisions, benchmark results, funding programs, policy stakeholders.

Estimated effort: ~4-8 hours. Modify each submit endpoint's Zod schema + handler. The source_check_verdicts write infrastructure already exists.

Component 2: PR Manifests for TableBase Changes

Since PG changes bypass git, agents write a manifest file to their PR branch documenting what they submitted:

// data/tablebase-manifests/2026-03-25-apart-research-personnel.json
{
  "table": "personnel",
  "entityId": "5WqgebIWFQ",
  "entityName": "Apart Research",
  "recordCount": 10,
  "submittedAt": "2026-03-25T18:12:09Z",
  "agentSession": "claude-opus-4-6-session-abc123",
  "verificationSummary": {
    "verified": 9,
    "contradicted": 0,
    "unverifiable": 1,
    "skipped": 0
  },
  "records": [
    {
      "personDisplayName": "Jaime Raldua Veuthey",
      "role": "CEO",
      "source": "https://apartresearch.com/impact",
      "verdict": "verified",
      "evidence": "Impact page lists 'Jaime Raldua Veuthey -- CEO'"
    },
    {
      "personDisplayName": "Esben Kran",
      "role": "Founder, Board Member",
      "source": "https://apartresearch.com/impact",
      "verdict": "verified",
      "evidence": "Listed as 'Board Member' on impact page; founder confirmed by multiple sources",
      "isFounder": true,
      "startDate": "2022"
    }
  ]
}

Gate check: New gate rule tablebase-manifest-coverage that:

• Detects if the PR branch submitted any TableBase records (by checking agent session logs or explicit manifest files)
• Fails if manifests are missing
• Warns if any records have skipped verification

What this gives you:

• Git-tracked audit trail for PG changes
• PR-reviewable summary of every TableBase submission
• Humans can see exactly what data an agent added before merging
• Manifest + PG data can be cross-validated

Component 3: PG Audit Log

Add a tablebase_audit_log table for complete change history:

CREATE TABLE tablebase_audit_log (
  id BIGSERIAL PRIMARY KEY,
  record_type TEXT NOT NULL,          -- 'personnel', 'grants', etc.
  record_id VARCHAR(10) NOT NULL,
  operation TEXT NOT NULL,             -- 'insert', 'update', 'delete'  
  old_data JSONB,                      -- null for inserts
  new_data JSONB NOT NULL,
  source_url TEXT,
  verdict TEXT,                        -- from verification
  evidence TEXT,
  agent_session_id TEXT,               -- links to agent_sessions table
  pr_number INTEGER,                   -- links to GitHub PR (nullable)
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_audit_log_record ON tablebase_audit_log(record_type, record_id);
CREATE INDEX idx_audit_log_session ON tablebase_audit_log(agent_session_id);
CREATE INDEX idx_audit_log_created ON tablebase_audit_log(created_at);

Every write to any TableBase table also writes to the audit log. This gives git-like history without changing the write path.

Dashboard integration: New "Recent TableBase Changes" section on the system health dashboard showing recent mutations, grouped by session and PR.

Component 4: FactBase Gate Rule (Source-Verification Coverage)

For YAML-based data (FactBase facts, entity descriptions), add a gate rule:

pnpm crux w validate gate --fix
# Now includes: source-verification-coverage

This rule:

1. Diffs packages/factbase/data/things/*.yaml against the base branch
2. Extracts new/modified facts
3. Cross-references each fact's source: URL against source_check_verdicts
4. Fails if any new fact has no corresponding verdict
5. Warns if any fact's verdict is older than 90 days

Implementation: Parse the YAML diff, extract fact IDs, query wiki-server verdicts API. ~4-6 hours.

Component 5: crux research Command (Verified Research)

One command replaces the 3-agent manual workflow:

pnpm crux research apart-research --website=https://apartresearch.com

Steps:

1. Fetches the website + key linked pages (team, about, impact)
2. Extracts structured claims using LLM (team members, metrics, dates, funding)
3. Immediately source-checks each claim against the page it came from
4. Outputs a structured research bundle:

# .claude/temp/research/apart-research/bundle.yaml
entity: 5WqgebIWFQ
researched_at: 2026-03-25T10:00:00Z
sources_fetched:
  - url: https://apartresearch.com/impact
    fetched_at: 2026-03-25T10:00:12Z
    status: 200
    content_hash: sha256:abc123

claims:
  - id: c1
    text: "Jaime Raldua Veuthey is CEO"
    source: https://apartresearch.com/impact
    evidence: "Listed as 'Jaime Raldua Veuthey -- CEO'"
    verdict: verified
    confidence: 0.95
    targets:
      - layer: personnel
        data: { role: CEO, roleType: key-person, source: "https://apartresearch.com/impact" }
      - layer: entity_yaml
        field: relatedEntries
        data: { type: person, relationship: related }

  - id: c2
    text: "5,000+ research sprint participants"  
    source: https://apartresearch.com/impact
    evidence: "Impact page says '3,000 research sprint participants'"
    verdict: contradicted
    corrected: { text: "3,000+ research sprint participants", value: 3000 }
    targets:
      - layer: factbase
        data: { property: sprint-participant-count, value: 3000, asOf: "2026-03" }

summary:
  total_claims: 47
  verified: 41
  contradicted: 3
  unverifiable: 3

Then crux enrich apply reads the bundle and writes to all layers — but only verified claims. Contradicted claims are logged. Unverifiable claims are flagged for human review.

Key insight: Research and verification happen in the same step, not as separate agent dispatches. The LLM sees the source page and checks the claim in one pass, which is both faster and more accurate than checking after the fact.

Component 6: Continuous Re-verification (Phase 4 Simplified)

The simplest version of "website-as-data-feed" doesn't need website snapshots. It just re-runs source-check on existing records periodically:

1. Daily cron: pnpm crux source-check recheck --needs-refresh --budget=5

   • Uses the existing nextCheckDue field in source_check_verdicts (Auto-recheck scheduler for verification verdicts #3021)
   • Prioritizes: contradicted > stale > never-checked

2. Verdict flip detection: If a verdict changes from verified to contradicted:

   • Auto-file a GitHub issue with the old value, new evidence, and source URL
   • Label: verification:contradicted
   • Include a suggested fix

3. Dashboard widget: "Recently Flipped Verdicts" on Source Checks (E2200) showing verdicts that changed in the last 7 days.

This captures ~90% of the value of full website monitoring with ~10% of the effort. Full website snapshots + structured diff detection (Discussion #2928) becomes a later phase.

Phasing

Phase	What	Effort	Impact
0	API-level verification enforcement (Component 1)	4-8h	Prevents ALL unverified TableBase writes going forward
1	FactBase gate rule (Component 4) + PG audit log (Component 3)	8-12h	Prevents unverified FactBase writes; enables TableBase change history
2	PR manifests (Component 2) + manifest gate check	6-8h	Makes TableBase changes reviewable in PRs
3	crux research command (Component 5)	16-24h	Replaces manual multi-agent research workflow
4	Continuous re-verification cron (Component 6)	4-8h	Detects when verified data becomes stale or contradicted

Open Questions

1. Should --skip-verification exist? It's pragmatic for bootstrapping/migration but creates a loophole. Options: (a) allow but log prominently, (b) allow but require human approval, (c) don't allow at all.

2. What verdict threshold is acceptable? Should we allow unverifiable claims to be written? Many source URLs are JS-rendered (Verification pipeline can't verify JS-rendered pages #3008) and return empty content. Blocking these would prevent ~20% of legitimate data additions.

3. Should contradicted claims be auto-rejected or flagged? Auto-rejection is safer but may block legitimate updates where the source has changed. Flagging requires human review bandwidth.

4. How should this interact with auto-update? The auto-update pipeline already fetches news and modifies pages. Should it also be required to verify its changes? This would slow it down significantly.

5. Manifest file lifecycle: Should manifests be permanent (growing archive in data/tablebase-manifests/) or ephemeral (deleted after merge, with PG audit log as the permanent record)?

6. Retroactive verification: 94% of existing TableBase records have no verdicts. Should we batch-verify existing data before enforcing the new rules, or grandfather existing records and only enforce for new submissions?

Relationship to Existing Discussions

• Unified Verification Architecture — Data Layer & Verification Decisions (March 2026) #2950 (Unified Verification Architecture): This proposal builds directly on the unified source_check_evidence + source_check_verdicts tables decided there
• Websites as Data Feeds: Extracting Structured Facts from Org Websites into TableBase/FactBase #2928 (Websites as Data Feeds): Component 6 is a simplified version; full implementation becomes a future phase
• Comprehensive Source-Check Expansion: TableBase, FactBase, and WikiBase Verification #3119 (Source-Check Expansion): Components 1 and 4 directly address the "expand coverage to all record types" goal
• External Research Patterns: STORM, Loki, RefChecker, GPT-Researcher — Experiments for Content Pipeline #3087 (External Research Patterns): Component 5 incorporates the Loki-style "decompose → query → verify" pattern
• Per-type research playbooks for entity research #3004 (Research Playbooks): Component 5 could be extended with per-type playbooks
• Content pipeline creates pages without footnote citations #3005 (Pages Without Citations): Component 5's verified research would feed into page creation with proper citations
• Auto-recheck scheduler for verification verdicts #3021 (Recheck Scheduler): Component 6 implements the scheduler that reads nextCheckDue

[OAGr]

Status (April 2026): Not started. The verification infrastructure is mature (unified tables, LLM source-checking, CLI orchestration), but verification remains post-hoc. No write-time verification gates exist. This is an architectural direction discussion — keeping open for future consideration.

[OAGr]

Superseded by Discussion #3993 (Source-Check → Sourcing Rename — Implementation Plan).

The architecture and naming decisions in this discussion have been consolidated into the 7-phase sourcing rename plan. Key changes: routes become /sourcing, tables rename to record_sources/source_verdicts, components standardize on SourcingDot/SourcingBadge.

Phase 0: #4001 | Phase 1: #4002