No Cap

Is it fact or cap? AI fact-checking with receipts.

No Cap is an evidence-first fact-checking application. It does not ask an LLM to guess whether a claim is true — it retrieves real-world evidence, ranks sources by credibility, and has an LLM reason over that evidence, with a separate adversarial review pass before any verdict is finalized.

Evidence first. AI second.

Status

MVP-complete across all ten build phases (see docs/ for architecture, evals, cost, and security detail). Every phase's checkpoint was run for real — including real end-to-end runs against OpenAI + Tavily, not just mocked tests. See Definition of Done below for an item-by-item status.

What it does

Submit a claim, paste an article, give it a URL, or upload an audio/video file. The pipeline extracts atomic, independently-verifiable claims; classifies them (opinions are never fact-checked); generates multiple search queries and retrieves real evidence via Tavily; has an LLM reason over that evidence only (never its own "knowledge"); runs an adversarial critic pass that actively tries to prove the verdict wrong; and, when the critic disagrees or the claim is unusually complex, escalates to a stronger model for final adjudication. Every citation is re-verified as a genuine, verbatim excerpt of something actually fetched before the verdict is shown — nothing is fabricated.

Stack

• Frontend: Next.js 16 (App Router), TypeScript, Tailwind CSS v4, shadcn/ui (@base-ui/react under the hood, not Radix), Framer Motion, Lucide React, React Hook Form, Zod, TanStack Query, next-themes — package manager: pnpm.
• Backend: Python, FastAPI, uv, Pydantic v2, SQLAlchemy + Alembic, LangChain, LangGraph, FFmpeg, faster-whisper.
• Data: Supabase (Postgres, Storage, Auth) — hosted, not self-hosted; a local sqlite file is used automatically when no Supabase project is configured.
• Search: Tavily or Exa.
• Observability: Langfuse (optional — the app runs fully with LANGFUSE_ENABLED=false; cost/token/call metrics are tracked independently of it either way).

Every LLM/search/transcription provider sits behind an interface (LLMProvider, SearchProvider, TranscriptionProvider, and the future-facing RetrievalProvider) with a Mock* implementation, so the whole stack — ingestion, the full LangGraph pipeline, the API, the frontend, the eval harness — runs and is testable at zero cost by default.

Project structure

no-cap/
├── apps/
│   ├── web/     Next.js frontend
│   └── api/     FastAPI backend (uv-managed) — the LangGraph pipeline lives here
├── evals/       Evaluation suite (FEVER, AVeriTeC, HoVer, custom)
├── docs/        ARCHITECTURE.md, EVALS.md, COST_OPTIMIZATION.md, SECURITY.md
├── .env.example
├── docker-compose.yml
└── package.json

apps/api and evals are members of a single uv workspace (see the root pyproject.toml) — evals imports directly from apps/api/app rather than duplicating provider/schema code. See docs/ARCHITECTURE.md for the full pipeline diagram and design decisions.

Getting started

Prerequisites

• Node.js 22+, pnpm (corepack enable or npm i -g pnpm)
• Python 3.12+, uv
• FFmpeg (only required once audio/video ingestion is exercised)

Backend

cp .env.example apps/api/.env   # defaults to mock providers — no API keys needed
uv sync --directory apps/api
uv run --directory apps/api uvicorn app.main:app --reload --port 8000

Visit http://localhost:8000/api/health — it reports which providers are active (mock by default).

Run tests / lint:

uv run --directory apps/api pytest
uv run --directory apps/api ruff check app tests

Apply database migrations (runs automatically against the local sqlite fallback the first time, but explicit is safer after pulling schema changes):

cd apps/api && uv run alembic upgrade head

Frontend

cd apps/web
cp .env.local.example .env.local
pnpm install
pnpm dev

Visit http://localhost:3000 for the app, http://localhost:3000/evals for the evaluation dashboard.

Google sign-in (optional)

Fact-checking works fully anonymously with no setup — sign-in is an optional add-on (same posture as Langfuse/PII elsewhere in this app), via Google as a Supabase Auth provider. To enable it:

1. Google Cloud Console: create an OAuth 2.0 Client ID (Web application); authorized redirect URI https://<project-ref>.supabase.co/auth/v1/callback.
2. Supabase Dashboard → Authentication → Providers → enable Google, paste in that Client ID/Secret; add http://localhost:3000/auth/callback (and your deployed URL's equivalent) to the allowed redirect URLs.
3. Set NEXT_PUBLIC_SUPABASE_URL/NEXT_PUBLIC_SUPABASE_ANON_KEY in apps/web/.env.local and SUPABASE_JWT_SECRET in apps/api/.env (all three are on Supabase's own API settings page).

Once set, a "Sign in" button appears in the header; signing in stores a profile row in public.users and attributes future fact-checks to that user (fact_checks.user_id) — anonymous fact-checks remain fully supported either way.

Both at once

From the repo root:

npm run dev:api    # or: pnpm run dev:api
npm run dev:web

Evaluations

uv run python -m evals.run --dataset custom                # mock, zero cost
uv run python -m evals.run --dataset custom --limit 5 --provider openai --model gpt-4o-mini

Run from the repo root. See docs/EVALS.md for dataset format, metrics, and real checkpoint results.

API

Endpoint	Purpose
POST /api/fact-check	Submit a claim or pasted text ({claim} or {text, title?})
POST /api/fact-check/url	Submit a URL
POST /api/fact-check/upload	Submit an audio/video file
GET /api/fact-check/{id}	Poll for status / get the full result
GET /api/sources/{id}	Look up a single source
GET /api/evals, /api/evals/{id}	Evaluation runs and their results
GET /api/metrics	Real aggregate cost/token/call metrics
GET /api/health	Which providers are active

All three POST endpoints return 202 {id, status} immediately and process in the background — poll GET /api/fact-check/{id} for the result. Rate limited (10/min per IP by default, RATE_LIMIT_FACT_CHECK_PER_MINUTE).

Configuring real providers

Everything defaults to mock (LLM_PROVIDER, SEARCH_PROVIDER, TRANSCRIPTION_PROVIDER) so local development costs nothing. To use real providers, set the corresponding API key and provider name in apps/api/.env — see .env.example for the full list. No model name or provider is ever hard-coded outside app/core/config.py and app/core/model_registry.py.

Documentation

• docs/ARCHITECTURE.md — full pipeline diagram, provider abstractions, data model, real bugs found and fixed along the way.
• docs/EVALS.md — running the harness, dataset format, metrics, real checkpoint results.
• docs/COST_OPTIMIZATION.md — model routing, deterministic bypasses, measured real costs.
• docs/SECURITY.md — SSRF protection, prompt-injection boundaries, rate limiting, what's still open.

Definition of Done

Walked item by item against the original spec's Section 42 checklist.

Done and verified (not just implemented — each of these was actually run, not just written):

• All five input types (claim, article/text, URL, audio upload, video upload) work end to end, with automated tests for each through the real API + real FFmpeg.
• URL content is classified (YouTube / direct media / webpage), audio and video are transcribed (real faster-whisper, real FFmpeg).
• Atomic claims are extracted and classified; opinions are never marked false.
• Multiple search queries are generated per claim (Tavily, real search verified in checkpoints); sources are ranked by a credibility model.
• Evidence is extracted with full provenance and a hard anti-fabrication gate (verbatim-substring check, verified with a fake LLM that fabricates a quote — it never survives).
• Real evidence-based fact-checking, adversarial review, and conditional final adjudication all ran for real (real OpenAI + real Tavily) across three separate checkpoints (Phases 5, 8, 9).
• Citations are validated before being shown; results and uploaded media are stored (Supabase-ready schema; uploads currently land on local disk — see Deferred).
• Langfuse traces the workflow when enabled; token/cost usage is tracked either way.
• Models, search provider, and transcription provider are all env-configurable, verified by grep to have zero hard-coded model strings outside the config layer.
• Vector search is correctly NOT required for the MVP; the architecture supports future pgvector activation (PgVectorRetrieval stub in place).
• The custom evaluation dataset runs end to end, both mock (zero cost) and real-provider.
• Security tests exist and were run for real: SSRF protection (including a DNS-rebinding case), rate limiting, and — most notably — a real prompt-injection resistance test against a live model.
• README and docs/ are complete.
• Local development requires zero paid API usage by default.
• A USD cost-budget limiter (per-fact-check and process-wide hourly caps, independent of the request-count rate limiter) guards against runaway spend; see app/core/cost_budget.py and docs/RISK_FRAMEWORK_ALIGNMENT.md.
• The groundedness guard that checks a verdict's reasoning against its cited evidence now also covers the adversarial critic's own notes field before it's rendered in the UI, not just the final verdict.
• CSP and other security-headers middleware is wired in (app/core/security_headers.py).
• URL/content fetches are cached in-memory with a TTL (app/core/url_cache.py) — see Caching below.
• Google sign-in via Supabase Auth: app/services/auth.py verifies the session JWT, POST /api/auth/session stores the user in public.users, and fact_checks.user_id is attributed when signed in. Optional — see "Google sign-in" above.

Deferred, named explicitly rather than silently dropped:

• PDF/image input — not built. Section 8 lists these as "eventually."
• Video frame/vision analysis — VisionAnalysisProvider interface exists and is documented as a stub; not implemented. The audio-transcript branch of video processing is fully real.
• pgvector activation — deliberately not enabled; PgVectorRetrieval raises NotImplementedError until the evidence corpus is large enough to justify it, per the spec's own guidance.
• FEVER / AVeriTeC / HoVer — loaders are real and functional, but no converted dataset file has actually been run through them yet (they're large third-party downloads, not bundled).
• Regression/pairwise comparison view in the evals dashboard — the comparison table supports eyeballing multiple runs side by side, but there's no dedicated diff feature between two specific runs.
• Real magic-byte MIME sniffing on uploads — extension + declared Content-Type only; judged not worth a libmagic system dependency given FFmpeg already fails safely on malformed input.
• Supabase Storage for uploads — StorageService is implemented but not wired into the upload path; files currently land on local disk. Wiring it in means also wiring the download-back-to-local step FFmpeg needs, which hasn't been tested against a real bucket.
• Caching (Section 35) — URL/content fetches are cached (see Done above); search-result caching and full repeated-claim/verdict caching are still unbuilt — see docs/COST_OPTIMIZATION.md.

What the next phase after MVP should be

In priority order: (1) extend caching to repeated claims/queries (search results, and eventually full verdicts for a repeated/similar claim — the URL-fetch layer is done, this is the bigger remaining piece and needs a staleness policy for temporal claims first); (2) wire Supabase Storage for uploads now that the local-disk path has proven the rest of the flow works; (3) run a real FEVER/AVeriTeC/HoVer sample once a converted dataset file exists, to get real benchmark numbers rather than just the custom set; (4) frame-level video vision analysis, behind the already-stubbed VisionAnalysisProvider interface; (5) enable pgvector once the evidence corpus from real usage is large enough to matter.

License

Not yet decided.