Resource/Sources System: Page Duplication, Missing Automation, and Pipeline Gaps

[OAGr]

Overview

The resource/sources system has accumulated significant duplication, dead code, and missing automation. This discussion catalogs all the problems and proposes a unified cleanup plan organized around one architectural principle.

Core Principle: Resources First, Source Checks Second

The entire resource system should be understood as two layers with a strict dependency:

Layer 1 — Resource Pipeline (build the library): Fetch URLs, cache content, generate summaries. This runs proactively when resources are created or periodically refreshed. After it completes, every resource should have cached full text (or a documented reason why not), a summary, and enrichment metadata.

Layer 2 — Source Check (verify claims against the library): Compare wiki claims against resources that already exist in our database. Source-check should never fetch URLs itself — it reads from citation_content and nothing else. If a resource hasn't been cached yet, that's a Layer 1 problem, not something source-check should work around.

Resource created → resource-verify job → fetch + cache content → resource-enrich job → summary
                                              ↓
                                    citation_content table (the library)
                                              ↓
Source check runs → reads cached content → verifies claims → done (no HTTP fetches)

Today these layers are tangled: source-check has its own fetcher, resource-verify discards content, enrichment is manual, and the content cache is empty. Every problem below traces back to this architectural confusion.

---

Problem 1: Three Overlapping Public Resource Pages

We have three separate URLs that all try to show the same thing — a browsable list of resources:

URL	Title	What it shows	Status
/sources	"Sources"	Tabbed view: Resources table + Publications table, summary stats	Working, full-featured
/resources	"Resources"	Same resource table as the Resources tab on /sources, plus type-breakdown stats	Working, duplicate
/wiki/E874	"External Resources"	Uses <ResourcesIndex> — a stub component never ported from Astro	Broken

/sources is a superset of /resources. The Resources tab on /sources renders the exact same ResourcesTable component with the same ResourceRow type. /resources just adds some extra stat cards (type counts, "Enriched" count) that /sources doesn't have.

/wiki/E874 is a dead page — ResourcesIndex is listed in the stub components array in mdx-components.tsx and renders a placeholder warning. It has never worked in the Next.js version.

Additionally, there are four wiki MDX pages in the sources area:

Wiki ID	Path	Title	Component	Purpose
E1049	content/docs/sources/sources-overview.mdx	"Sources"	<SourcesOverviewContent />	Stats + links to /resources and /publications
E1043	content/docs/sources/factbase-resources.mdx	"Resources"	<FBResourcesContent />	Internal-style resource table with fetch status
E1044	content/docs/sources/factbase-publications.mdx	"Publications"	<FBPublicationsContent />	Internal-style publications table
E874	content/docs/tools/resources.mdx	"External Resources"	<ResourcesIndex /> (stub)	Broken

The "Resources" concept is spread across 7 different URLs (/sources, /resources, /resources/[id], /publications, /wiki/E1049, /wiki/E1043, /wiki/E874) with overlapping content and inconsistent naming.

Solution

• Delete E874 (content/docs/tools/resources.mdx) — broken stub. Redirect /wiki/E874 to /sources.
• Consolidate /resources into /sources — merge the extra stat cards (type breakdown, enrichment pipeline progress) into the /sources Resources tab. Then redirect /resources → /sources.
• Keep /resources/[id] as the detail page for individual resources.
• Review E1049/E1043/E1044 — these wiki MDX pages duplicate the /sources App Router pages. Consider redirecting to the App Router versions.
• Remove ResourcesIndex from the stub list in mdx-components.tsx once E874 is deleted.
• Fix navigation — the header nav under "Data" links to /sources, but /resources, /wiki/E874, /wiki/E1043, and /wiki/E1049 are all independently reachable. After consolidation, ensure nav only links to canonical URLs.

---

Problem 2: The Content Cache Is Empty — resource-verify Discards Fetched Content

This is the root cause of most pipeline problems. The resource-verify job handler fetches URLs to check liveness but throws away the HTML after computing a hash. The citation_content table — designed to cache full text — is essentially empty.

Production data (from a recent investigation):

Metric	Value
Total resources	5,326
fetched_at IS NOT NULL	563 (10.6%)
Resources with cached content in citation_content	~0

Even the 563 resources marked as "fetched" have zero cached content. fetched_at is set by resource-verify, but the content is discarded.

Impact on Layer 2 (source-check):

• Source-check tries citation_content first → empty → falls back to cold HTTP fetch
• 26% of URLs fail on re-fetch (403, paywall, bot-blocking)
• No Firecrawl fallback in the source-check fetcher
• Every source-check run re-fetches the same URLs

This violates the core principle: source-check should read from a populated library, not do its own fetching.

Solution

Make resource-verify cache content in citation_content when it successfully fetches a URL. This is the single highest-leverage change — it populates the library that both enrichment and source-check depend on.

The handler should also persist fetch_status, last_fetched_at, and content_hash back to the resource record (currently these are returned in JobHandlerResult.data but never written to the DB).

Related issues: #3449, #3452, #3457, #3472. PR: #3455.

---

Problem 3: Enrichment Pipeline Is Entirely Manual

The enrichment pipeline that generates summaries is a 3-stage manual CLI process:

Stage 1: pnpm crux resources fetch-all           → fetches URLs, caches content
Stage 2: pnpm crux resources classify             → LLM classifies type/purpose (Haiku batch)
Stage 3: pnpm crux resources deep-enrich submit   → LLM generates summaries (Sonnet batch)

Each stage must be run manually, in order, with separate polling for batch completion. When a new resource is added or verified, nothing triggers enrichment.

Current state: 5,326 resources, only 1,861 have summaries (35%).

Why batch-only is a problem

The Anthropic Batch API gives a 50% discount but:

• Requires manual submission, polling (30s intervals for hours), and result download
• All-or-nothing — can't process one resource at a time
• Only cost-effective at scale (5K+), but new resources trickle in one at a time
• Results take hours, not seconds

Solution: resource-enrich job handler + automatic chaining

Following the "resources first" principle, enrichment should be part of the resource pipeline — triggered automatically, not manually. Create a resource-enrich job type that:

1. Takes { resourceId, url } as params
2. Reads cached content from citation_content (populated by the fixed resource-verify — see Problem 2)
3. Calls Claude (standard API, not batch) for classification + summarization in a single pass
4. Writes summary, key_points, tags, importance_score, resource_subtype back to the resource
5. Updates enrichment_status to 'enriched'

Automatic chaining: When resource-verify completes successfully with status === 'reachable', it enqueues a resource-enrich job. The full automated pipeline becomes:

resource created → resource-verify → cache content + persist status
                                         ↓
                                   resource-enrich → summary, tags, importance
                                         ↓
                              resource fully populated in DB
                                         ↓
                        source-check reads cached content (no fetching)

Cost: ~1-2¢ per resource (standard API) vs ~0.5¢ (batch). The 2x markup is worth it for full automation.

The existing batch pipeline can remain as a bulk backfill tool for processing thousands at once, but the per-resource job handler is the primary path for new resources.

Merging classify into enrich

The classify stage (classify.ts) generates resource_subtype, resource_purpose, and context_note using Haiku. But the enrich stage also generates resource_purpose and context_note, overriding classification results. The only unique classify output is resource_subtype and sub_table.

For the per-resource resource-enrich handler: combine classification and enrichment into a single Sonnet call. One LLM call per resource, not two. The batch pipeline can keep separate stages for cost optimization if desired.

---

Problem 4: Three Separate Source-Fetching Implementations

Three independent systems fetch web content with no shared code:

System	File	Function	Used by
Source fetcher	crux/lib/search/source-fetcher.ts	fetchSource()	Enrichment pipeline (fetch-all.ts)
Source-check fetcher	crux/lib/source-check/source-fetcher.ts	fetchSourceContent()	Fact-checking, source verification
Resource-verify handler	crux/lib/job-handlers/resource-verify.ts	Raw fetch()	Job worker

All three detect paywalls and soft 404s, check HTTP status codes, and handle timeouts — but with different implementations. The source fetcher has domain-aware strategies (Firecrawl, Wayback, forum APIs); the other two don't.

Under the "resources first" principle, only the resource pipeline should fetch URLs. Source-check should read from citation_content only.

Solution

1. Make resource-verify use the shared source-fetcher (fetchSource()) instead of rolling its own fetch() — this gives it domain-aware strategies, Firecrawl, Wayback Machine support
2. resource-verify caches content in citation_content as a side effect (see Problem 2) — this eliminates the need for the separate fetch-all CLI stage
3. Simplify source-check's fetcher to a pure citation_content reader with no HTTP fallback. If the cache misses, report "resource not cached" — don't try to fetch
4. The fetch-all CLI command becomes a bulk backfill tool (re-cache everything), not the primary caching path

End state: one fetching implementation (source-fetcher), used by one system (resource-verify), writing to one cache (citation_content). Source-check only reads.

---

Problem 5: Enrichment Status Not Visible in Stats or Dashboards

The /api/resources/stats endpoint returns:

{
  "totalResources": 5326,
  "withMetadata": 1861,
  "fetched": 563,
  "orphanedCount": "???",
  "byType": { "web": 3981, "paper": 465, ... }
}

Missing: No breakdown by enrichment_status (pending, fetched, classified, enriched, reviewed). No way to see pipeline progress — how many resources are stuck at each stage. No visibility into how full the content cache is.

Solution

Add byEnrichmentStatus and contentCacheStats to the stats endpoint:

{
  "byEnrichmentStatus": {
    "null": 2800,
    "pending": 200,
    "fetched": 400,
    "classified": 65,
    "enriched": 1861,
    "reviewed": 0
  },
  "contentCacheStats": {
    "withFullText": 0,
    "withMetadataOnly": 14,
    "uncached": 5312
  }
}

Surface this on the consolidated /sources page as a pipeline progress indicator. This makes it immediately obvious whether the resource pipeline is healthy and how much of the library is populated for source-check.

---

Problem 6: fetch_status vs enrichment_status Confusion

The resource schema has two status-like fields:

• fetch_status — the internal dashboard assumes values "full" | "metadata-only" | "unfetched", but nothing in the codebase consistently writes these values
• enrichment_status — the enrichment pipeline's state machine (pending → fetched → classified → enriched → reviewed)

These overlap confusingly. resource-verify doesn't write to either (Problem 2). fetch-all writes to enrichment_status but not fetch_status. The UpdateResourceFetchStatusSchema exists in api-types.ts but its usage is unclear.

Solution

Audit whether fetch_status serves a distinct purpose from enrichment_status. If it's about content availability (full text vs metadata-only vs nothing), consider making it a computed field derived from citation_content state. If it's dead, deprecate it.

With the resource pipeline properly caching content (Problem 2), the content availability question can be answered by querying citation_content directly rather than maintaining a separate status field.

---

Problem 7: Dead / Tactical Code

File	Purpose	Action
crux/resource-enrichment/fix-enrichment-status.ts	One-time backfill for missing enrichment_status — tactical fix for a batch size bug	Delete
ResourcesIndex stub in mdx-components.tsx	Placeholder for never-ported Astro component	Remove with E874 (Problem 1)
crux/resource-enrichment/fetch-all.ts	Bulk URL fetching + content caching	Keep as backfill tool — but primary caching moves to resource-verify (Problem 2)
crux/lib/source-check/source-fetcher.ts	Source-check's independent fetcher	Simplify to read-only — no HTTP fallback (Problem 4)
/source/[id]/page.tsx	Redirect from old URL pattern	Keep (backward compat)
/sources/publications/[id]/page.tsx	Redirect from old URL pattern	Keep (backward compat)

---

Current State of the Pipeline (What Happens Today)

Resource created (via citation indexing or manual add)
    ↓
resource-verify job (job queue)
    → Fetches URL, checks liveness
    → Computes content hash
    → Sets fetched_at on resource
    → ❌ DISCARDS the fetched HTML
    → ❌ Does NOT cache in citation_content
    → ❌ Does NOT persist fetch_status
    → ❌ Does NOT trigger enrichment
    ↓
citation_content table: EMPTY
    ↓
source-check runs (manual)
    → Tries citation_content cache → miss
    → Falls back to cold HTTP fetch (its own fetcher)
    → 26% fail (paywall, 403, bot-blocked)
    → No Firecrawl, no domain-aware strategies
    ↓
enrichment pipeline (manual CLI, 3 stages)
    → fetch-all: re-fetches all URLs AGAIN
    → classify: Haiku batch (manual submit + poll)
    → enrich: Sonnet batch (manual submit + poll)
    → Only 35% of resources have been through this

Target State

Resource created
    ↓
resource-verify job (automatic)
    → Uses shared source-fetcher (domain-aware, Firecrawl, Wayback)
    → ✅ Caches full text in citation_content
    → ✅ Persists fetch_status, content_hash, last_fetched_at
    → ✅ Enqueues resource-enrich job on success
    ↓
resource-enrich job (automatic, chained)
    → Reads cached content from citation_content
    → Single Sonnet call: classify + summarize + tag
    → ✅ Writes summary, key_points, tags, importance_score
    → ✅ Sets enrichment_status = 'enriched'
    ↓
citation_content table: POPULATED
    ↓
source-check runs
    → Reads from citation_content ONLY (no HTTP fetches)
    → If cache miss: reports "resource not cached" → signals need for resource pipeline run
    → Pure verification: does this cached content support this claim?

---

Action Items (Priority Order)

These are ordered by dependency — each step enables the next:

Phase 1: Fix the resource pipeline (enables everything else)

1. Make resource-verify cache content + persist results — the single highest-leverage change. Populates citation_content, writes fetch_status/content_hash to resource records. (Problems 2, 4)

2. Create resource-enrich job handler — combined classify+enrich in one Sonnet call, triggered automatically after resource-verify. (Problems 3, 6)

3. Add enrichment/cache stats to API — byEnrichmentStatus and contentCacheStats in /api/resources/stats. (Problem 5)

Phase 2: Simplify source-check (now that the library is populated)

4. Strip source-check fetcher to read-only — remove HTTP fallback, read from citation_content only. If cache miss, report it instead of fetching. (Problem 4)

Phase 3: Clean up the UI

5. Consolidate resource pages — merge /resources into /sources, delete E874, add pipeline progress indicator, fix nav. (Problems 1, 7, 9)

6. Audit fetch_status field — document or deprecate. (Problem 6)

7. Delete dead code — fix-enrichment-status.ts, ResourcesIndex stub. (Problem 7)

---

Questions for Discussion

• Should we keep the batch enrichment pipeline alongside the per-resource worker, or phase it out once per-resource is working?
• What's the acceptable per-resource cost for automatic enrichment? (~1-2¢ standard API vs ~0.5¢ batch)
• Should /sources be the canonical URL, or rename to something clearer like /library?
• For source-check read-only mode: should a cache miss be a hard error or a warning? (i.e., does source-check refuse to verify uncached resources, or skip them with a note?)

[OAGr]

Resource Pipeline v2: End-to-End Agent Workflow & Future Concerns

This comment expands the discussion beyond the original 9 problems (mostly resolved) into a broader view of what the resource pipeline needs to become. The framing: an agent should be able to submit URLs, get structured results back, and use those results for source-checking — all without manual intervention.

---

The Target Agent Workflow

AGENT RESEARCHING A TOPIC
    ↓
1. Submit URLs
   POST /api/resources/suggest { urls: [...], entityId?: "..." }
   ← { batchId: "batch-abc", results: [{ url, resourceId, contentStatus }] }
   System auto-enqueues resource-ingest jobs for missing/stale URLs
    ↓
2. Poll for completion (while doing other work)
   GET /api/resources/suggest/batch-abc/status
   ← { pending: 2, completed: 7, failed: 1,
       results: [{ url, resourceId, status, summary, error? }] }
   Agent can start using completed resources immediately
    ↓
3. Use cached content for source-checking
   Source-check reads from citation_content — no HTTP fetches
   If cache miss → returns "not_cached" → agent knows to wait/re-submit
    ↓
4. Get verification results
   GET /api/verifications/by-entity?entityId=anthropic
   ← { verified: 12, contradicted: 1, unverifiable: 3, unchecked: 5 }

Today's state: Step 1 exists (/api/resources/suggest) but doesn't trigger ingestion. Step 2 doesn't exist — no batch tracking. Step 3 works but agents hit not_cached walls with no recovery path. Step 4 has no query endpoint.

---

Issue A: Agent Batch Submission & Polling API

Problem: An agent submitting 10 URLs has no way to track them as a group. It would need to call createJob() 10 times individually, store 10 job IDs, and poll each separately.

Proposed solution: Extend /api/resources/suggest to:

1. Auto-enqueue resource-ingest jobs for URLs with contentStatus: "missing" or "stale"
2. Return a batchId linking all the jobs
3. Add GET /api/resources/suggest/:batchId/status that returns aggregated status across all jobs in the batch
4. Include enrichment results (summary, tags) in the response once resource-enrich completes

The job infrastructure already supports batch creation (createJobBatch()) and dedup keys. The missing piece is the orchestration layer that ties suggest → job creation → status polling into one workflow.

Partial completion: When 7/10 URLs are done, the agent should be able to start using those 7 while waiting for the remaining 3. The status endpoint should return per-URL results as they complete.

---

Issue B: Wayback Machine Fallback in resource-ingest

Problem: resource-ingest does its own raw fetch() and doesn't use the shared source-fetcher (fetchSource()), so it doesn't get Wayback Machine fallback, Firecrawl integration, forum API support, or DOI resolution.

Current state: The shared source-fetcher already has:

• Wayback Machine fallback for 404/410 responses
• Dead domain detection (FHI Oxford, FTX Fund → wayback-only strategy)
• Forum APIs (LessWrong, Alignment Forum, EA Forum → GraphQL)
• DOI resolution (Nature, Science, Springer → CrossRef)
• Firecrawl for bot-blocked domains (OpenAI, Reuters, NYT)

Fix: Make resource-ingest use fetchSource() instead of rolling its own HTTP call. This was identified as part of Problem 4 but only the source-check side was fixed — resource-ingest still has its own fetch implementation.

---

Issue C: Content Type Handling

Problem: Many resources are PDFs, videos, or structured data — not HTML pages.

Content type	Current handling	What should happen
HTML pages	Fetched, htmlToText(), cached	✅ Works
PDFs	Fetched as binary, htmlToText() mangles them	Extract text (pdftotext, Firecrawl PDF mode)
YouTube videos	Handled by source-fetcher (transcript API)	✅ Works if using shared fetcher
Other video/audio	Not handled	Transcript extraction (Whisper API?)
GitHub repos	Fetched as HTML (renders README)	Could use GitHub API for structured data
JSON/API responses	htmlToText() strips angle brackets	Skip HTML stripping, store as-is

The resource-ingest handler now conditionally applies htmlToText() based on content-type (PR #3506), but PDF extraction is still missing.

---

Issue D: Multi-Page Resources (Sites vs. Pages)

Problem: A single "resource" like quantifieduncertainty.org has multiple relevant pages (About, Team, Research, Blog posts). The current model is one URL = one resource.

Options:

1. Parent-child resources: A "site" resource with child "page" resources. Parent holds domain-level metadata, children hold per-page content.
2. Resource groups: Tag multiple resources as belonging to the same site. No hierarchy, just grouping.
3. Domain aggregation view: Keep the flat model, add a "by domain" view that groups resources by hostname.

Option 3 is lowest effort and may be sufficient. The data model already captures individual pages as separate resources — the missing piece is browsing them grouped by site.

Agent workflow implication: When an agent submits anthropic.com/research, should the system also discover and ingest anthropic.com/about, anthropic.com/responsible-scaling-policy, etc.? This would be a "site crawl" mode for resource-ingest.

---

Issue E: Duplicate Resource Detection

Problem: The same paper might exist at arxiv.org, semanticscholar.org, the author's site, and the journal. We could have 4 resource records for one paper.

Detection signals:

• contentHash match (same content at different URLs)
• DOI match (same paper, different hosts)
• Title+author fuzzy match
• Canonical URL detection (<link rel="canonical">)

Impact: Wasted LLM costs (enriching the same paper 4 times), confusing citation counts, fragmented source-check results.

---

Issue F: Resource Staleness & Periodic Re-ingestion

Problem: Content changes over time. A policy document gets updated, a blog post gets corrected, an organization restructures their site. We cache content once and never refresh.

The contentLifecycle field exists on resources (immutable/versioned/evergreen/ephemeral) but nothing acts on it.

Proposed: A scheduled sweep (daily or weekly) that re-ingests resources based on lifecycle:

• ephemeral (news, tweets): re-ingest monthly, expect many to be dead
• evergreen (org pages, policy docs): re-ingest quarterly
• versioned (papers with revisions): re-ingest when contentHash changes
• immutable (published papers): never re-ingest unless marked dead

---

Issue G: Cost Tracking & Budget Guards

Problem: Each resource-enrich call costs ~1-2¢. A full re-enrichment of 5,300 resources costs ~$50-100. But there's no cost tracking per resource. The job table has a costUsd column but it's not populated.

Proposed:

• resource-enrich handler should estimate and record cost in job.costUsd
• Budget guards on bulk operations (e.g., --budget=50 flag on enqueue commands)
• Dashboard showing cumulative LLM spend by job type

---

Issue H: Source-Check Cannot Self-Recover from Cache Misses

Problem: When source-check hits a URL that's not in citation_content, it returns errorType: 'not_cached' and stops. The agent gets a dead-end error with no recovery path.

Current state: Three commands (factbase-source-check, verify-entity, add-citations) have their own local fetchSourceContent() implementations that still do direct HTTP fetching — they bypass the read-only source-fetcher. This creates inconsistency: some pipelines silently fetch on cache miss, others fail.

Proposed: When source-check encounters not_cached:

1. Auto-enqueue a resource-ingest job for the URL (fire-and-forget)
2. Return not_cached to the agent with resourceId and jobId so they can poll
3. On the next source-check run, the content should be cached

This makes the pipeline self-healing: source-check never fetches directly, but it signals the resource pipeline to fill the gap.

---

Issue I: No Query API for Verification Status

Problem: Evidence and verdicts are stored in the DB, but there are no GET endpoints to query them. Agents cannot discover:

• "Which claims on page X are unverified?"
• "Which facts about entity Y have contradicted verdicts?"
• "What's the overall verification coverage for this entity?"
• "Which URLs failed verification due to not_cached?"

Proposed endpoints:

GET /api/verifications/by-entity?entityId=X
  ← { verified: 12, contradicted: 1, unverifiable: 3, unchecked: 5 }

GET /api/verifications/by-page?pageId=X
  ← { claims: [{ text, verdict, sourceUrl, confidence }] }

GET /api/verifications/failures?errorType=not_cached&limit=50
  ← { items: [{ recordId, sourceUrl, errorType, lastAttempt }] }

GET /api/verifications/stale?days=90
  ← { items: [{ recordId, lastVerifiedAt, verdict }] }

---

Issue J: Prioritized Ingestion

Problem: Not all resources are equally important. A paper cited by 40 wiki pages should be ingested and enriched before an obscure blog post cited once. The bulk enqueue command processes resources in arbitrary order.

Proposed: Priority scoring for resource-ingest jobs:

• Citation count (how many wiki pages reference this URL)
• Entity importance (resources linked to high-importance entities first)
• Recency (newer resources may be more time-sensitive)
• Source-check demand (URLs that are blocking active source-checks)

---

Issue K: Rate Limiting & Domain Politeness

Problem: Ingesting 5,000 URLs hits thousands of domains. Some rate-limit, some might ban our IP. The current pipeline has per-request timeouts but no per-domain throttling.

Proposed: Per-domain concurrency limits in the worker:

• Max 1 concurrent request per domain
• 2-second delay between requests to the same domain
• Domain-level backoff on 429 responses
• Respect robots.txt and Crawl-delay headers

---

Issue L: Resource Discovery & Suggestions

Problem: Wiki pages cite URLs that aren't in the resources table. There's citation indexing that creates resources from footnotes, but what about inline links? Or URLs in FactBase facts? Or resources an LLM could suggest based on entity context?

Proposed: When enriching an entity or page, the LLM could suggest additional resources:

• "This org also publishes an annual report at X"
• "This paper has a follow-up study at Y"
• "The primary source for this claim is Z"

These suggestions get added to a "suggested resources" queue for agent review.

---

Issue M: Three Separate Verification Pipelines

Problem: Three independent commands handle verification with different fetch strategies and output formats:

1. source-check-orchestrate — unified facts + records + entities (cache-only fetch)
2. factbase-source-check — fact-centric (has its own HTTP fallback)
3. verify-entity — entity-scoped combination

Impact: Agent confusion on which to use. Bugs need fixing in 3 places. Inconsistent verdicts.

Proposed: Consolidate into one pipeline with scope filters:

crux verify --scope=entity --id=anthropic
crux verify --scope=page --id=ai-safety
crux verify --scope=fact --id=fact-123
crux verify --scope=all --limit=50 --budget=5

---

Summary: Priority Order

Phase 1 — Make the pipeline self-healing (highest impact):

• (H) Source-check auto-enqueues ingestion on cache miss
• (B) resource-ingest uses shared source-fetcher (Wayback, Firecrawl, forum APIs)
• (I) Query endpoints for verification status

Phase 2 — Agent batch workflow:

• (A) Suggest endpoint triggers ingestion + returns batch ID for polling
• (J) Prioritized ingestion based on citation count / demand

Phase 3 — Content quality & coverage:

• (C) PDF and non-HTML content extraction
• (E) Duplicate detection
• (F) Periodic re-ingestion based on content lifecycle
• (L) Resource discovery / suggestions from LLM

Phase 4 — Scale & operations:

• (G) Cost tracking and budget guards
• (K) Per-domain rate limiting
• (M) Consolidate verification pipelines
• (D) Multi-page resource grouping

[OAGr]

Status (April 2026): Audit findings documented. Resource enrichment pipeline (crux/resource-enrichment/) and source-check orchestration exist. The page duplication and pipeline gap issues identified here are partially addressed but not fully resolved. Keeping open as a reference for resource system cleanup work.

[OAGr]

Issues Filed from This Audit

After reviewing the current codebase, most of the core pipeline problems (Problems 2-5, Issues A/B/H from the comment) have been addressed in the pipeline v2 work (March 2026). Specifically:

• Problem 2 (content cache empty): resource-ingest now caches in citation_content ✅
• Problem 3 (manual enrichment): resource-enrich job handler with automatic chaining ✅
• Problem 4 (three fetchers): source-check is cache-only with auto-enqueue on miss; resource-ingest uses shared source-fetcher ✅
• Problem 5 (no enrichment stats): byEnrichmentStatus and contentCacheStats added ✅
• Problem 7 (dead code): E874 deleted, ResourcesIndex stub removed, fix-enrichment-status.ts deleted ✅
• Issue A (batch workflow): suggest endpoint with batch tracking + polling ✅
• Issue B (shared source-fetcher): resource-ingest uses fetchSource() ✅
• Issue H (self-healing pipeline): auto-enqueue on cache miss ✅

Three remaining high-impact gaps have been filed as issues:

1. Consolidate duplicate source/resource wiki pages (E1043, E1044, E1049) #3662 — Consolidate duplicate source/resource wiki pages (E1043, E1044, E1049)

   • Problem 1 partial: App Router redirect done, but 3 wiki MDX stubs still duplicate /sources content. Dead resources-table.tsx component remains.

2. Add verification query API — by-entity, by-page, failures, stale #3665 — Add verification query API (by-entity, by-page, failures, stale)

   • Issue I: No GET endpoints for agents to discover unverified claims, contradictions, or cache-miss failures programmatically.

3. Audit and resolve fetch_status vs enrichment_status confusion on resources #3668 — Audit and resolve fetch_status vs enrichment_status confusion

   • Problem 6: Two overlapping status fields with unclear write semantics. Now that pipeline v2 maintains enrichment_status consistently, fetch_status may be dead weight.

Lower-priority items from the comment (Issues C/D/E/F/G/J/K/L/M) are not yet addressed but are less urgent — they represent scale/polish improvements rather than architectural gaps.

[OAGr]

Status Update — April 6, 2026

This discussion is effectively resolved. All 7 original problems plus the 3 filed follow-up issues have been addressed across a series of PRs.

Entity-Resources Cleanup (PRs #3936, #3943, #3951, #3965)

Replaced the fragmented source/resource system with a proper entity_resources join table:

Phase	PR	Status	What
1	#3936	Merged	entity_resources join table (schema, API, seed command)
2	#3943	Merged	Remove entity YAML sources field (3,768 lines deleted)
3	#3951	Merged	Org page integration, remove entity_recommended_resources + people-resources.yaml
4	#3965	Merged	Remove heuristic fallback, improve pub types, add sourceResourceId FKs

Table seeded: 4,182 rows in production (publisher + wiki citation passes).

Architecture Now

• entity_resources join table with authoredByEntity / isSubject boolean flags replaces:
  • Entity YAML sources field (deleted)
  • Heuristic domain-matching in org-data.ts (deleted, ~400 lines)
  • entity_recommended_resources table (dropped)
  • people-resources.yaml (deleted)
• Org pages query entity_resources directly — no more URL path sniffing
• sourceResourceId FK columns added to personnel, grants, funding_rounds, investments, equity_positions (backfill script ready)

Follow-up Issues (all resolved)

• Consolidate duplicate source/resource wiki pages (E1043, E1044, E1049) #3662 — Consolidate duplicate source/resource wiki pages ✅
• Add verification query API — by-entity, by-page, failures, stale #3665 — Add verification query API ✅
• Audit and resolve fetch_status vs enrichment_status confusion on resources #3668 — Audit fetch_status vs enrichment_status ✅

Remaining Minor Items

• Backfill sourceResourceId: SQL script ready at apps/wiki-server/scripts/backfill-source-resource-ids.sql
• Literature.yaml seed pass: pnpm crux entity-resources seed --source=literature adds 25 paper→org links

Recommend closing this discussion.

[OAGr]

Rename source-check → sourcing — Implementation Plan

TL;DR

The "source-check" system has three coexisting naming conventions (source-check, verification, SourceCheck), a name that implies post-hoc verification when it also handles write-time sourcing, and redundant sourceResourceId FK columns on 5 tables. This plan renames to "sourcing" with record_sources/source_verdicts as the canonical table names. Code migrates first, tables rename last (avoiding the broken view strategy). Phase 1 is a 20-file cleanup that finishes the incomplete March 2026 rename. The full rename is ~162 files across 7 phases.

Problem

The source-check system is the wiki's primary infrastructure for record-level provenance (linking structured data to their backing source URLs with verification verdicts). Three problems:

1. Three naming conventions coexist: routes use "source-check", shared helpers use "verification-*", components live in components/verification/ with SourceCheck* names
2. "Source-check" implies post-hoc verification, but the system also handles write-time sourcing, LLM discovery, and inline submission
3. sourceResourceId on 5 tables is redundant with source_check_evidence.resourceId

Current State

Asset	Current name	Target name	Refs
DB table	source_check_evidence	record_sources	~11 files
DB table	source_check_verdicts	source_verdicts	~17 files
Drizzle symbols	sourceCheckEvidence, sourceCheckVerdicts	recordSources, sourceVerdicts	~12 files
Deprecated aliases	verificationEvidence, verificationVerdicts	(remove)	2 exports
API route	/api/source-checks	/api/sourcing	1 mount
Deprecated API alias	/api/verifications	(remove)	1 mount
Wiki-server route	routes/source-check/source-checks.ts (1787 lines)	routes/sourcing/sourcing.ts	1 file
Shared helpers	verification-join.ts, verification-enforcement.ts, verification-schema.ts	sourcing-join.ts, sourcing-enforcement.ts, sourcing-schema.ts	~4 files, ~15 importers
API response fields	verificationVerdict, verificationConfidence, etc.	sourcingVerdict, sourcingConfidence, etc.	~7 frontend files
Frontend pages	app/source-checks/	app/sourcing/	6 files
Internal dashboards	entity-source-checks/, source-check-coverage/	entity-sourcing/, sourcing-coverage/	11 files
Component dir	components/verification/	components/sourcing/	3 files + ~22 consumers
CLI lib	crux/lib/source-check/ (30 files)	crux/lib/sourcing/	30 files
CLI commands	source-check-orchestrate, etc.	sourcing-orchestrate, etc.	~10 files
GitHub Actions	source-check.yml, source-check-recheck.yml	sourcing.yml, sourcing-recheck.yml	2 files
FK columns	sourceResourceId on 5 tables	(remove)	~9 files
tsconfig aliases	@wiki-server/source-checks-route	@wiki-server/sourcing-route	2 files

Proposed Approach

Code-first, tables-last (learned from red-team: ON CONFLICT breaks through PG views, so the original "rename tables first + backward-compat views" strategy is non-viable).

Order:

1. Finish the incomplete March 2026 rename (kill "verification" residue)
2. Migrate all code to new Drizzle symbol names (still pointing at old PG table names)
3. Rename frontend pages, components, CLI
4. Rename PG tables last (one-line ALTER TABLE RENAME, instant)
5. Remove deprecated aliases

Key Decisions

• Route path: /sourcing (not /records per Rename /source-checks to /records — unified record detail page #3923)

  • Alternatives: /records (too generic), /sources (collides with existing hub page)
  • Rationale: "Sourcing" communicates provenance specifically

• Code-first, tables-last (not tables-first with views)

  • Alternatives: Rename tables first + backward-compat views
  • Rationale: ON CONFLICT doesn't work through PG views. Red-team finding.

• Verdict unification (Unify verdict vocabularies: source-check vs citation pipeline #3447): Separate project

  • Rationale: Semantic change (stored data + LLM prompts) mixed with mechanical rename makes regressions undiagnosable

• discoveryMethod column: Separate PR

  • Rationale: Feature addition, not rename. Clean separation of intent.

• sourceResourceId removal: Phase 0 (before rename starts)

  • Rationale: Independent concern, reduces surface area for the rename

Implementation Phases

Phase 0: Remove sourceResourceId + finish March rename — S

Goal: Clean foundation. One consistent name ("source-check") before introducing "sourcing".

• Drop sourceResourceId from 5 tables (migration)
• Delete resolve-source-resource.ts
• Remove sourceResourceId from sync schemas, formatRow, ON CONFLICT in grants/personnel/investments/funding-rounds/equity-positions
• Rename verification-join.ts → source-check-join.ts
• Rename verification-enforcement.ts → source-check-enforcement.ts
• Rename verification-schema.ts → source-check-schema.ts
• Move components/verification/ → components/source-check/
• Rename 4 Next.js proxy routes from verification-*-proxy to source-check-*-proxy
• Remove deprecated /api/verifications route alias
• Remove deprecated verificationEvidence/verificationVerdicts Drizzle re-exports
• Rename API response fields: verificationVerdict → sourceCheckVerdict (server + ~7 frontend consumers)

Files: ~30. Risk: Low (mechanical, no DB table changes).
Quality gates: pnpm build, pnpm test, grep for verification in .ts/.tsx returns zero (excluding migration SQL).
Exit criteria: One name ("source-check") everywhere.

Phase 1: Drizzle symbol rename (code points at old tables) — S

Goal: All TypeScript uses new symbol names. PG tables unchanged.

• Rename sourceCheckEvidence → recordSources in schema.ts (still pgTable("source_check_evidence", ...))
• Rename sourceCheckVerdicts → sourceVerdicts in schema.ts (still pgTable("source_check_verdicts", ...))
• Update all ~12 files importing these symbols
• Rename API response fields: sourceCheckVerdict → sourcingVerdict (server + frontend)
• Verify: Temporarily remove re-exports, confirm TypeScript still compiles (catch masked bugs). Then remove them permanently.

Files: ~20. Risk: Low (TypeScript catches all broken refs).
Quality gates: pnpm build, pnpm test, grep -r "sourceCheckEvidence\|sourceCheckVerdicts" --type ts = zero.

Phase 2: Server route + shared helpers rename — S

Goal: /api/sourcing is canonical. Server code says "sourcing".

• Rename routes/source-check/source-checks.ts → routes/sourcing/sourcing.ts
• Rename exported type: SourceChecksRoute → SourcingRoute
• Mount /api/sourcing as primary, keep /api/source-checks as temporary alias
• Rename source-check-join.ts → sourcing-join.ts (from Phase 0)
• Rename source-check-enforcement.ts → sourcing-enforcement.ts
• Update tsconfig aliases
• Update crux wiki-server client: source-checks.ts → sourcing.ts

Files: ~15. Risk: Low (alias keeps old path working).
Quality gates: curl /api/sourcing/stats returns valid JSON.

Phase 3a: Frontend component rename — M

Goal: Components say "Sourcing" instead of "SourceCheck".

• SourceCheckDot.tsx → SourcingDot.tsx (+ re-export at old path)
• FactSourceCheckDot.tsx → FactSourcingDot.tsx
• source-check-status.ts → sourcing-status.ts
• Rename types: SourceCheckStatus → SourcingStatus, etc.
• Update ~22 consumer files
• Update mdx-components.tsx (keep old MDX component name as alias)

Files: ~28. Risk: Medium (many consumers, but mechanical).
Quality gates: pnpm build, spot-check sourcing dots render on org pages.

Phase 3b: Frontend page + dashboard rename — S

Goal: Users see /sourcing in the URL bar.

• Rename app/source-checks/ → app/sourcing/ (6 files)
• Rename app/internal/entity-source-checks/ → app/internal/entity-sourcing/ (9 files)
• Rename app/internal/source-check-coverage/ → app/internal/sourcing-coverage/ (2 files)
• Add Next.js permanent redirects from old paths
• Update getSourceCheckHref() → getSourcingHref()
• Update nav links, sidebar, wiki-nav.ts

Files: ~20. Risk: Low (redirects catch broken links).
Quality gates: /sourcing loads, /source-checks redirects.

Phase 3c: CLI + workflows rename — M

Goal: CLI says crux fb sourcing, workflows say sourcing.yml.

• Rename crux/lib/source-check/ → crux/lib/sourcing/ (30 files)
• Rename CLI commands + add deprecated aliases in crux.mjs
• Rename GitHub Actions workflow files
• Rename auto-update, tablebase, validation references
• Update all internal imports

Files: ~45. Risk: Medium (largest single phase, conflict risk).
Quality gates: pnpm build, pnpm crux fb sourcing --help works, old crux fb source-check still works via alias.

Phase 4: DB table rename + cleanup — S

Goal: PG tables match code. Clean break.

• Migration: ALTER TABLE source_check_evidence RENAME TO record_sources (instant)
• Migration: ALTER TABLE source_check_verdicts RENAME TO source_verdicts (instant)
• Migration: Rename indexes (idx_sce_* → idx_rs_*, idx_scv_* → idx_sv_*)
• Update schema.ts: change pgTable("source_check_evidence", ...) → pgTable("record_sources", ...)
• Remove /api/source-checks route alias
• Remove CLI command aliases
• Remove Next.js redirects (optional — keep for SEO)
• Update entity YAML slugs for dashboard pages
• Update internal documentation, CLAUDE.md, .claude/rules/

Files: ~15. Risk: Low (all code already uses new names, this is just the PG rename).
Quality gates: grep -r "source_check_evidence\|source_check_verdicts\|sourceCheck\|source-check" --type ts = zero (excluding migration SQL). Full test suite. Production health check after deploy.

Scope Cuts

• Verdict vocabulary unification (Unify verdict vocabularies: source-check vs citation pipeline #3447): Separate project. Different risk profile.
• source column rename on records: Not doing. Too many tables, marginal value.
• Monolith decomposition of 1787-line sourcing.ts: Orthogonal. Easier after rename.
• Record Inspector page (Rename /source-checks to /records — unified record detail page #3923): Separate feature.
• discoveryMethod column addition: Separate PR.

Quality & Verification Infrastructure

Area	What
Tests	Existing tests updated (mechanical). No new tests needed. Full suite after each phase.
Migration verification	After each phase: grep for old patterns. Phase 4 final audit: zero source_check/sourceCheck/SourceCheck/verification refs in non-SQL .ts/.tsx files.
Integration smoke test	After Phase 2: curl /api/sourcing/stats. After Phase 3b: load /sourcing page. After Phase 4: health check.
UI verification	After Phase 3a: sourcing dots render on org pages, grants table, entity profile.
Deploy steps	Phase 4 only: migration runs on startup (standard). No new env vars.
Documentation	Phase 4: CLAUDE.md, .claude/rules/, internal docs.
Monitoring	Phase 4: audit entry in .claude/audits.yaml for 30-day post-deploy check.

Risks & Mitigations

Risk	Severity	Mitigation
Previous rename needed 6 fix-ups for missed refs	High	Code-first strategy means TypeScript catches all. Post-phase grep audit. Temporarily remove re-exports to find masked bugs.
Merge conflicts in 45-file CLI PR (Phase 3c)	High	Time during merge window. Announce 24h before. ./ws refresh after merge.
API response field names cross HTTP boundary	High	Explicitly tracked. Server + frontend renamed in same phase.
ON CONFLICT with views	Eliminated	Code-first, tables-last. No views needed.
Aliases never removed (calcification)	Medium	Audit entry with deadline. Build-time warning after 8 weeks.
Is this worth 162 files of effort?	Medium	Phase 0 alone (20 files) fixes the worst inconsistency. Phases 1-4 are incremental — can stop after any phase.

Open Questions

1. Should we keep /source-checks permanent redirects for SEO, or remove after 6 months?
2. MDX component names: Keep SourceCheckDot as MDX alias forever, or update MDX pages?
3. API field names: Rename sourcingVerdict etc., or keep verificationVerdict to avoid HTTP-boundary churn?

Rejected Approaches

• A: Fix verification residue only (20 files): Standardizes to "source-check" but doesn't achieve the conceptual improvement of "record_sources"/"sourcing". Adopted as Phase 0 — the foundation before the full rename.
• Tables-first with views: ON CONFLICT doesn't work through PG views. Blocking issue found in red-team.
• C2: Record Inspector: Monolith decomposition is valuable but orthogonal to naming.
• C3: Annotation Layer: YAGNI — only one annotation type exists.

Red Team Log

Round 1 findings (3 agents):

Technical critic:

• BLOCKING: ON CONFLICT breaks through PG views → Reversed phase order (code first, tables last)
• BLOCKING: API response field names are hidden HTTP-boundary coupling → Added explicit tracking
• SIGNIFICANT: verification-enforcement.ts and verification-schema.ts missing from inventory → Added
• SIGNIFICANT: Dynamic imports evade grep → Added as checklist item
• MINOR: Index rename rationale was wrong (ON CONFLICT uses expressions, not names) → Corrected

Scope critic:

• Full rename is 162 files for a naming preference, not a bug fix → Addressed honestly in Risks table. Phase 0 alone fixes the worst problem. Phases are incremental — can stop after any.
• discoveryMethod is scope creep → Removed from plan
• Backward-compat views are overkill → Eliminated (code-first strategy)
• Phase 0 lint guard is busywork → Removed
• 8-week alias retention is unnecessary for internal API → Shortened, with audit deadline

UX critic: N/A (internal infrastructure, no user-facing UX changes beyond nav label)