RhetoriQ

RhetoriQ is an evidence-first narrative investigation system. It detects public narrative signals, retrieves source material, maps how language changes and spreads, and produces reports whose material claims point back to inspectable evidence.

The product deliberately distinguishes first observed in the available dataset from true origin and does not treat correlation as proof of coordination.

flowchart TB
    subgraph Edge[Restricted public edge — final proof only]
        U([User]) --> B[Browser]
        DNS[Route 53 DNS] --> ALB[HTTPS ALB / Ingress]
        ACM[ACM certificate] -.- ALB
        B -->|HTTPS pages / API / SSE| ALB
        ALB -->|/| FE[React / Nginx]
        ALB -->|/api| API[FastAPI]
    end
    subgraph Core[EKS application namespace]
        API -->|accept request| PG[(PostgreSQL / pgvector)]
        PG ==>|transactional outbox| OUT[Outbox publisher]
        OUT ==>|versioned events| K[(Kafka KRaft)]
        REG[Apicurio schemas] -.- K
        K ==>|requested| IW[LangGraph investigation worker]
        K ==>|raw / enriched| FL[Flink stream job]
        FL ==>|processed / signals| DW[Document + signal workers]
        DW -->|canonical writes| PG
        IW -->|receipts + artifacts| PG
        IW --> GATE{{Claim checks + publication gate}}
        GATE -->|cited report or limitation| PG
        K ==>|projection topics| PW[Projection workers]
        PW --> ES[(Elasticsearch)]
        PW --> NG[(Neo4j)]
        PW --> V[(MiniLM vectors in pgvector)]
        API -->|validated reads| ES
        API -->|explained paths| NG
        API -->|complete-response cache| RD[(Redis cache)]
        API -->|durable history / workspace| PG
    end
    subgraph Sources[Approved acquisition]
        SX[SearXNG] --> WEB[Public records + permitted pages]
        IW -->|bounded tools| SX
        IW -->|primary APIs / fetch| WEB
    end
    classDef actor fill:#eef2ff,stroke:#4f46e5,color:#1e1b4b
    classDef service fill:#e0f2fe,stroke:#0284c7,color:#082f49
    classDef durable fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef derived fill:#fef3c7,stroke:#b45309,color:#78350f
    classDef gate fill:#fce7f3,stroke:#be185d,color:#831843
    classDef trust fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class U,B actor
    class FE,API,OUT,IW,FL,DW,PW,SX,WEB service
    class PG,K durable
    class ES,NG,V,RD derived
    class GATE gate
    class DNS,ALB,ACM,REG trust

Solid arrows show direct requests and writes; thick arrows show asynchronous events; dotted lines show trust relationships. See the full architecture for runtime processing, deployment, recovery, the complete legend, and the lifecycle walkthrough.

The repository includes the product, Kafka/Flink pipeline, B5 projections, B6 Helm chart, guarded EKS Terraform states, and deployment scripts.

• FastAPI endpoints for ingestion, trending topics, investigations, timelines, graphs, mutations, receipts, and reports.
• GDELT DOC 2.0 ingestion for news discovery.
• Hacker News ingestion through the public Algolia API.
• Direct HTTP retrieval of canonical pages for evidence enrichment.
• A durable LangGraph research runtime with budgets, leases, checkpoints, idempotent actions, replay, SSE progress, and a deterministic publication gate.
• Self-hosted SearXNG discovery plus GDELT, Hacker News, canonical HTTP, internal-corpus, and an isolated Playwright adapter for local research only.
• Federal Register first-party research with policy-aware receipts, retries, pagination, deduplication, and visible limitations.
• Apache Kafka KRaft and Apicurio Registry with 12 versioned primary topics and matching DLQs, a transactional outbox, idempotent consumers, controlled replay, and Kafka-only investigation dispatch.
• SQLite-backed development storage and PostgreSQL/pgvector production persistence, with optional Redis capabilities.
• A React and TypeScript investigation interface with a live graph, research rail, evidence gate, and replay controls.
• Flink document processing and narrative signals plus B5 Elasticsearch, Neo4j, MiniLM/pgvector, and Redis projections behind acceptance-gated flags.
• Production containers, committed PostgreSQL migrations, and CI checks for backend, frontend, schemas, documentation, and migration compatibility.
• A shared Helm chart for kind and EKS, three isolated Terraform states, GitHub OIDC publishing to immutable ECR images, and bounded smoke, evidence, recovery, and teardown scripts.

See the documentation index, B6 operations, and roadmap for implementation and operating details.

RhetoriQ is agent-led and source-policy-first, not crawler-first:

1. A user question starts a bounded investigation; the LangGraph investigator selects live broad-web search, internal-corpus recall, canonical-page retrieval, or an approved primary-source API for each evidence gap.
2. Search results and API records are discovery leads, not automatically evidence.
3. RhetoriQ retrieves a canonical source page when permitted and needed to create an evidence record, then preserves receipts and limitations.
4. The event pipeline processes accepted documents and signals; additional scheduled RSS/Atom and event-stream connectors are planned for recurring monitoring.

A website, post, transcript, or official record is a source. An API, feed, or HTML fetch is the transport used to retrieve it.

Current source status is documented in DATA_SOURCES.md.

rhetoriq/
|-- backend/
|   |-- agents/       # planning, retrieval, synthesis, and receipts
|   |-- api/          # FastAPI route modules
|   |-- models/       # shared Pydantic contracts
|   |-- services/     # ingestion, retrieval, analysis, and persistence
|   `-- tests/
|-- frontend/         # React, TypeScript, and Vite application
|-- docs/             # design and operating documentation
|-- deploy/helm/      # kind and EKS application chart
|-- infra/terraform/  # isolated AWS bootstrap, foundation, and platform states
|-- SYSTEM_DESIGN.md
`-- README.md

cd backend
python -m venv ..\.venv
..\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
uvicorn main:app --reload

The API is available at http://127.0.0.1:8000; interactive documentation is at /docs.

The running OpenAPI document is the authoritative endpoint reference. The API includes health, ingestion, narratives, trending, investigations, research events/replay, evidence search, and provenance-path routes. Accepted ingestion and investigation work is Kafka-only; the API commits an outbox record and returns without executing the job synchronously.

cd frontend
npm install
npm run dev

Vite normally serves http://127.0.0.1:5173. Routes are /, /dashboard, and /investigation/:id. The production Nginx image reads PUBLIC_API_BASE_URL at container startup; local Vite development uses VITE_API_BASE_URL.

The root Compose stack supplies PostgreSQL/pgvector, Kafka, Apicurio Registry, SearXNG, topic initialization, the outbox publisher, role-scoped workers, Flink, API, and frontend:

$env:POSTGRES_PASSWORD="<local-secret>"
$env:SEARXNG_SECRET="<local-secret>"
docker compose up --build -d
docker compose ps

See Architecture for research and service boundaries and Operations for startup, replay, and recovery.

pytest backend/tests
cd frontend
npm run build

Settings are loaded from backend/.env when present.

Connector credentials are configured only for approved deployments. Reddit access, commercial search APIs, and licensed news products require terms and retention review before production use.