Skip to content

Home

Jump to bottom
Nikolai Sachok edited this page Jun 24, 2026 · 4 revisions

Strata-RAG Wiki

Strata-RAG is an eval-first, debuggable Retrieval-Augmented Generation engine over a heterogeneous, multi-format document corpus. It is written to be read as a tutorial on production RAG — heavily commented, study-grade — not just run.

It assembles the pieces a real document-intelligence system needs: pluggable source adapters, a tiered relevance classifier, ingestion observability (a dry-run manifest + a chunk inspector), a Qdrant + HNSW vector index, hybrid retrieval (dense + BM25) fused with Reciprocal Rank Fusion, a cross-encoder re-ranker, a structured metadata sidecar for exact aggregation, an eval harness (Recall@K / Precision@K / MRR / nDCG + LLM-as-judge faithfulness), and layered prompt-injection and secret/PII leak defenses.

It runs out of the box with no key and no real data: it ships a tiny synthetic, fully fictional sample corpus, and embeddings + retrieval are local. Only enrichment, generation, and the judge need an LLM backend.

Navigation

  • Architecture — the ingest → retrieve → generate pipeline, and the two-query-class design (semantic retrieval vs structured aggregation) that shapes the whole engine — taught as why this design, with the failure mode each stage prevents and where it sits vs enterprise scale.
  • Design-Decisions — the centerpiece: every major choice (vector store, embedder, reranker, hybrid+RRF, chunking, PII backend, top-k vs top-p, LLM-judge, the open-core plugin seam) as Decision → Why → Alternatives → Tradeoff → When you'd choose differently → How it scales.
  • Evaluation — the eval methodology: Recall@K / nDCG / MRR for retrieval, LLM-as-judge faithfulness for generation, why those two are kept strictly separate, what each metric's blind spot is, and how you'd harden it for production.
  • Engineering-Notes — debugging war stories told as teaching cases: a contaminated index, a structural leak a regex can't see, a cross-lingual retrieval miss, and a confidently mis-extracted attribute.
  • Red-Teaming — adversarial prompt-injection testing: obfuscation that evades a regex scanner, the two metrics (scanner-evasion vs end-to-end ASR), and the centerpiece teaching case — red-teaming your own success-oracle to a design that's sound by construction.

The one-paragraph mental model

Ingest does two things in parallel: it builds a semantic index (Qdrant) for "what is this about?" questions, and a structured metadata sidecar (SQLite) for "how many / which set?" questions. Retrieval over the semantic index is hybrid (dense + sparse), fused, then reranked. A vector top-k can't count or intersect, and a metadata table can't reason about meaning — so the engine keeps both, and (from Phase 2) routes each question to the right one. Everything is measured: retrieval against a golden set, generation against an LLM judge, and the privacy boundary against deterministic + semantic leak gates.

Roadmap at a glance

  • Phase 1 (this repo) — text ingest, dual indexes, hybrid retrieval + rerank, eval, guardrails, leak gates.
  • Phase 1.5 — multimodal: caption-then-embed images, manifest-discovered icons, image leak coverage in CI.
  • Phase 2 — agentic chatbot: query router, multi-turn agent, thin Streamlit UI, optional OpenAI-compatible endpoint.
  • Phase 3 — multilingual retrieval A/B and Prometheus + Grafana observability.

See the issue tracker and milestones for the breakdown.

Clone this wiki locally