Fraud Gauntlet

A system for validating government-issued IDs, extracting their fields into structured data, and flagging fraud or compliance issues in a form that plugs into existing notary and loan-origination workflows.

The project is split into three independent services that talk to each other over HTTP. Each lives in its own directory, has its own dependencies, and can be developed, tested, and deployed on its own.

┌─────────────┐       HTTP        ┌──────────────┐       HTTP        ┌──────────────┐
│  Frontend   │ ────────────────► │   Backend    │ ────────────────► │    Vision    │
│  (Next.js)  │                   │  (FastAPI)   │                   │  (FastAPI)   │
│   :3000     │ ◄──────────────── │    :8000     │ ◄──────────────── │    :9000     │
└─────────────┘    JSON / files   └──────────────┘    JSON / files   └──────────────┘
     UI                             orchestrator                       ML / CV / LLM

---

frontend/ — user interface

Purpose. The human-facing surface. Users upload an ID image (and optionally a loan document or selfie), watch the verification run, and see the structured result: extracted fields, authenticity score, cross-document match table, compliance findings with rule citations.

Stack. Next.js (App Router) + TypeScript + Tailwind.

Responsibilities.

• Collect files and metadata from the user.
• Call the backend API.
• Render the structured JSON response as a readable report.
• Surface errors and validation problems clearly.

Does not. Run OCR, call ML models directly, or implement compliance logic. It is a thin client over the backend's API.

---

backend/ — orchestrator and rules engine

Purpose. The single HTTP API that the frontend talks to. It accepts uploads, coordinates work, applies deterministic business and compliance rules, shapes the final response, and persists whatever needs to be persisted.

Stack. Python 3.11, FastAPI, Pydantic v2, asyncpg (Supabase / Postgres), httpx, structlog.

Responsibilities.

• Expose the public REST API (/api/v1/extract/id, /verify/id, /verify/cross-doc, /compliance/check, /health).
• Validate uploads (size, type, schema).
• Call the vision service for anything requiring OCR, computer vision, or an LLM.
• Apply pure-Python business rules: field normalization, fuzzy name matching, date/expiration checks, out-of-state rules, notary-commission validation, rule citations.
• Define and enforce the public Pydantic schemas — the contract the frontend builds against.
• Redact/hash PII before any log or metric write.

Does not. Import OpenCV, PaddleOCR, DeepFace, MediaPipe, or any LLM SDK. All model work is delegated to vision.

---

vision/ — ML, CV, and LLM service

Purpose. Everything that needs a model. OCR on ID images, layout/security-feature authenticity checks, face match, liveness detection, and an LLM reasoning agent for nuanced cross-document judgements.

Stack. Python 3.11, FastAPI, PaddleOCR, OpenCV, Pillow, DeepFace, MediaPipe, Anthropic SDK.

Responsibilities.

• POST /ocr/id — extract structured fields + per-field confidence from an ID image.
• POST /authenticity — score an ID for tampering; return labeled flags with bounding boxes.
• POST /face-match — similarity between ID photo and a selfie.
• POST /liveness — detect spoofing in a selfie burst or video.
• POST /reason/cross-doc — advisory LLM pass over (id_fields, doc_fields) with structured output.
• Keep model weights loaded and warm; expose readiness per subsystem on /health.

Does not. Store user data, enforce compliance rules, or talk to the frontend directly. It is a stateless inference service behind a stable HTTP contract.

---

How they fit together

1. A user uploads an ID through the frontend.
2. The frontend sends the image to the backend (POST /api/v1/verify/id).
3. The backend validates the request, then calls the vision service over HTTP (POST /ocr/id, POST /authenticity) to get the model-derived data.
4. The backend combines those results with its own rule engine (compliance checks, cross-document comparison, normalization), shapes the final response against its public Pydantic schemas, and returns JSON.
5. The frontend renders that JSON as a report for the user.

The backend is the only service the frontend knows about. The vision service is the only one that loads models. Each boundary is HTTP + JSON, so any service can be mocked, replaced, scaled, or moved to a different machine without changing the others.

The single seam

The backend reaches vision through one file: backend/app/services/vision_client.py. It has a mock mode (USE_MOCK_VISION=true) that returns fixture JSON from backend/tests/fixtures/vision/, so the backend and frontend can be developed end-to-end without the vision service running at all.

Configuration that wires them together

Variable	Where	Purpose
NEXT_PUBLIC_API_BASE_URL	frontend .env	Where the frontend finds the backend (e.g. http://localhost:8000)
VISION_BASE_URL	backend .env	Where the backend finds vision (e.g. http://localhost:9000)
USE_MOCK_VISION	backend .env	If true, backend serves fixture responses instead of calling vision
CORS_ORIGINS	backend .env	Allowed frontend origins
DATABASE_URL	backend .env	Supabase / Postgres URL (direct :5432 for Uvicorn, pooled :6543 for serverless)
STORE_VERIFICATIONS / STORE_MEDIA / REDACT_STORED_PII	backend .env	Persistence toggles (see backend/.env.example)

Changing a URL is the only thing that differs between running all three on one laptop, running them on one VM, and running them on three separate hosts.

---

Running it end-to-end (local)

1. Provision the Supabase database (once)

The backend persists every /verify result (and, if STORE_MEDIA=true, the raw uploaded images as BYTEA) to a Postgres database.

1. Create a project at supabase.com.
2. Project → Connect → copy the connection string:
   • Direct (port 5432) — use this for long-lived Uvicorn workers (the default local dev setup).
   • Pooled (port 6543, pgBouncer transaction mode) — use this for serverless-style deployments.
3. Paste it into backend/.env as DATABASE_URL (see backend/.env.example). The verifications table + indexes are created on first boot — no migrations to run.

Prefer not to use Supabase? Any Postgres 14+ works. Locally: docker run -p 5432:5432 -e POSTGRES_PASSWORD=pw postgres:16 and point DATABASE_URL at postgresql://postgres:pw@localhost:5432/postgres.

2. Start the three services

Three terminals:

# 1. Vision (models + OCR + LLM + PDF417 barcode)
cd vision
uv sync
uv run uvicorn app.main:app --reload --port 9000

# 2. Backend (API + rules + Supabase persistence)
cd backend
uv sync --extra dev
cp .env.example .env          # then paste your Supabase DATABASE_URL
uv run uvicorn app.main:app --reload --port 8000

# 3. Frontend (UI)
cd frontend
npm install
npm run dev

Then open the frontend at http://localhost:3000. Requests will flow frontend → backend → vision, and each verification will land as one row in your Supabase verifications table (inspect via Supabase's Table Editor).

To run without the vision service, set USE_MOCK_VISION=true in backend/.env and skip step 1.

To run without persistence at all, set STORE_VERIFICATIONS=false — the /verifications/* replay endpoints will return 503, but /verify still works.

---

Repository layout

.
├── frontend/   # Next.js app — user interface
├── backend/    # FastAPI service — public API, rules, orchestration
├── vision/     # FastAPI service — OCR, CV, face match, LLM agent
├── CLAUDE.md   # (top-level notes)
└── README.md   # this file

Each service has its own CLAUDE.md describing its internal conventions, scope, and boundaries.