Skip to content

Design Decisions

Jump to bottom
Nikolai Sachok edited this page Jun 18, 2026 · 1 revision

Design Decisions

This is the page to read if your question is "why this path, and not Y?" Every major choice in Strata-RAG is written up in the same shape:

Decision → Why → Alternatives considered → The tradeoff → When you'd choose differently → How it scales to enterprise.

The goal is to teach the reasoning, so the choices transfer to your own corpus. A decision with no stated alternative is just a feature; a decision with its rejected alternatives is engineering judgement. Where a choice composes with the larger pipeline, Architecture has the context; where it's about measuring the choice, Evaluation does.

A recurring meta-principle worth naming up front, because it drives several of these: the model proposes, a deterministic layer enforces. Embeddings, LLM judges, NER detectors, and rule advisors are all probabilistic proposers; rules, joins, and policies are the deterministic enforcers. Most of the decisions below are an instance of drawing that line in the right place.

Vector store — Qdrant (swapped from Chroma)

  • Decision. A standalone Qdrant server as the ANN index (the original prototype used embedded Chroma).
  • Why. An enterprise-style engine wants a real ANN server: a process you point multiple services at, inspect over HTTP (the dashboard), tune (explicit HNSW knobs), and that scales past in-process memory. The index is an HNSW graph with M / ef_construct / ef_search exposed in config.py.
  • Alternatives considered.
    • Chroma (embedded, in-process) — simplest; perfect for a notebook, but it's not the shape of a production index (no server to introspect or share).
    • pgvector (vectors as a Postgres column) — superb when your data already lives in Postgres and you want one transactional store; weaker pure-ANN ergonomics/perf at large scale.
    • Pinecone (fully-managed) — zero-ops, but a hosted dependency, ongoing cost, and your data leaves your boundary (a non-starter for a private corpus).
    • Weaviate / Milvus (self-hosted, distributed-capable) — powerful, but heavier to operate than a single-node teaching engine warrants.
  • The tradeoff. Qdrant buys a real ANN server (tunable HNSW, payload filters, HTTP introspection) for one docker compose up of operational cost — more than embedded Chroma, far less than a managed cloud or a distributed cluster.
  • When you'd choose differently. Notebook/throwaway → Chroma. Data already in Postgres at modest scale → pgvector (one store, transactional). No ops appetite / want managed → Pinecone. Billions of vectors / multi-node → Milvus, Weaviate, or Qdrant Cloud.
  • How it scales. Managed Qdrant Cloud or a multi-node cluster; quantization + on-disk payloads; per-tenant collections or payload-partitioning for isolation. The single-node docker compose here is the seam where managed ANN picks up.

Embedding model — all-mpnet-base-v2 (local default; MiniLM as the A/B baseline)

  • Decision. Default to local all-mpnet-base-v2 (sentence-transformers, 768-dim); keep all-MiniLM-L6-v2 (384-dim) as the measured baseline.
  • Why. Local keeps the private corpus on-device (no third-party send), and mpnet gives strong retrieval quality. Crucially, the engine doesn't assert mpnet is best — it ships a measured A/B (the collection name is derived from the model, so two indexes coexist; --dense-only isolates the embedder) so the choice is evidence, not vibes (see Evaluation §2).
  • Alternatives considered.
    • all-MiniLM-L6-v2 (384-dim) — ~2× faster/cheaper, lower quality ceiling; the A/B baseline.
    • Multilingual (paraphrase-multilingual-mpnet, multilingual-e5, BGE-M3) — required the moment retrieval must cross languages. This is roadmap (Phase-3 A/B), not yet run — and a monolingual embedder silently caps recall at its own language (Engineering-Notes #3).
    • Hosted OpenAI text-embedding-3-* — often higher quality, but the corpus leaves your boundary, plus per-call cost and a network dependency.
  • The tradeoff. Quality × dimensionality/latency × data residency. mpnet trades MiniLM's speed for accuracy; local trades OpenAI's possible quality edge for keeping private data on-device.
  • When you'd choose differently. Latency/cost-bound at scale → MiniLM (and measure the gap, don't assume it). Cross-lingual corpus → a multilingual embedder (and prove it on a non-EN golden slice). Already all-in on a hosted stack and data-residency is fine → OpenAI.
  • How it scales. Run the same A/B on the real corpus; re-eval as new models ship. Note: a re-embed is the only full-reindex trigger — the collection name encodes the model, so two models never collide.

Cross-encoder reranking (vs none, vs LLM rerank)

  • Decision. A cross-encoder (ms-marco-MiniLM-L-6-v2) re-scores the fused candidates; on by default, toggleable.
  • Why. Dense + BM25 are bi-encoders — they embed the query and each document separately, so the query never "sees" the document during scoring (fast, but lossy). A cross-encoder feeds (query, chunk) together through a transformer and outputs one relevance score — far more accurate ordering. Retrieve-then-rerank: cheap bi-encoders fetch ~20 candidates, the expensive cross-encoder scores only those → corpus-scale recall + pair-scale precision.
  • Alternatives considered.
    • No rerank — fewer moving parts, lower precision; the fused top-k goes straight to the generator (the default in many tutorials).
    • LLM-as-reranker — highest quality, but a per-query LLM call → latency + cost + a fresh injection surface on untrusted chunk text.
  • The tradeoff. A second model load + ~20 cross-encoder scores per query buys a markedly better final ordering — cheap precision. An LLM reranker buys a little more quality for a lot more cost.
  • When you'd choose differently. Ultra-low-latency and the embedder already ranks well → skip it (RAGEVAL_USE_RERANK=false). Quality-critical and budget allows → an LLM reranker (with injection guards on the chunk text).
  • How it scales. A stronger reranker (e.g. bge-reranker-large) on a GPU; cache (query, chunk) scores; tune candidate_k against the latency budget.

Hybrid dense + BM25 + RRF (vs dense-only)

  • Decision. Run dense ANN and sparse BM25, fuse with Reciprocal Rank Fusion.
  • Why. Dense captures meaning (synonyms, paraphrase — "citrus" finds "lemon") but blurs exact tokens (rare names, IDs, codes); BM25 nails exact terms but misses paraphrase. Each covers the other's blind spot. RRF fuses the two ranked lists using rank position only (1/(k+rank) summed) — because a cosine score and a BM25 score are on incomparable scales, so you cannot just add the raw numbers. Items ranked high by both retrievers rise.
  • Alternatives considered.
    • Dense-only — simpler, the tutorial default; but it silently fails on exact-term queries (a code or rare ID the embedding blurs away).
    • Weighted score fusion — needs per-retriever score normalization/calibration; fragile across queries and models.
    • Learned fusion — more infrastructure than a teaching engine warrants.
  • The tradeoff. One extra retriever + a BM25 index built at startup, for robustness on the exact-term queries dense-only drops. RRF is parameter-light (one constant k) and robust.
  • When you'd choose differently. Corpus is pure prose with no codes/IDs/rare names → dense-only may suffice (but measure it with --dense-only). You have reliable score calibration → weighted fusion can edge out RRF.
  • How it scales. Use Qdrant's native sparse vectors instead of an in-memory BM25 rebuilt from a scroll_all at startup — the current approach is fine for a demo/medium corpus and is flagged in retrieve.py as the explicit scale seam.

Chunking strategy + overlap (character-window, size 800 / overlap 150)

  • Decision. A pure character-window chunker with overlap.
  • Why chunk at all. One vector per whole document is a blurry average of every topic in it — retrieval can't pull "the theme paragraph" out of a multi-page doc. Small chunks give each passage its own precise vector. Why overlap: a hard cut orphans an idea that straddles the boundary; overlapping the tail of one chunk with the head of the next means a straddling idea survives intact in at least one chunk. The cost is a little duplication — cheap and worth it.
  • Alternatives considered.
    • Token-based chunking — aligns to the model's real context budget (more correct), needs a tokenizer.
    • Semantic / recursive chunking — split on headings/sentences for better boundaries; more complex.
    • Whole-doc / no chunking — loses retrieval precision entirely.
  • The tradeoff. The character-window chunker is pure, dependency-free, trivially unit-testable, and readable — deliberately chosen for a teaching engine. The principle (size + overlap) is identical to a token chunker; only the unit (chars vs tokens) differs.
  • When you'd choose differently. Production with a hard token budget → token chunking. Structured docs with clear headings → semantic/recursive. Very long coherent passages → larger chunks.
  • How it scales. Token-aware + structure-aware chunking, with per-doc-type chunk policies.

PII backend — regex default, optional Presidio NER (dual-backend + comparison harness)

  • Decision. Split PII into a policy (keep-or-redact, backend-agnostic) and a pluggable detector: regex by default, optional Presidio (spaCy NER) — and ship a comparison harness that runs both under the same policy.
  • Why. The keep/redact policy ("keep published/role-based contacts like support@, redact personal data") must NOT change when the detector changes — so detection and policy are cleanly separated, mirroring the embeddings local/openai factory. The regex default is zero-deps, no model download, deterministic, emails-only → the public demo and the fast test suite run out of the box. Presidio swaps in as a richer detector (PERSON/PHONE/IBAN/CREDIT_CARD) behind the unchanged policy.
  • Alternatives considered.
    • Regex-only — high precision on structured patterns, but blind to free-text PII (a name in prose).
    • NER-only / always-on — forces a heavy spaCy dependency + model download on everyone, including the zero-setup demo.
    • A hosted PII API — sends data off-box, which is wrong for a privacy layer.
  • The tradeoff (the teaching point). Cheap deterministic regex = high precision on structured patterns, blind to free-text; NER = broad coverage at the cost of a heavier, probabilistic backend that also false-positives (it tags some non-name words as PERSON). The comparison harness (pii_compare.py) quantifies exactly this on the same corpus under the same policy — with masked output (entity type + length + offset, never raw PII) so it's safe on a real corpus — and in doing so proves the policy is backend-agnostic.
  • When you'd choose differently. Structured/known PII shapes only → regex. Free-text names/phones in prose → Presidio (en_core_web_lg for production accuracy). Regulated + on-prem → Presidio local, never a hosted API.
  • How it scales. Presidio _lg (or a domain-tuned recognizer); a recognizer registry per data-class; the keep/redact policy promoted to governed config.

Top-k retrieval + optional absolute rerank-score floor (NOT top-p)

  • Decision. A fixed top-k context budget, plus an optional absolute floor on the rerank score (off by default). Explicitly not a top-p / nucleus cutoff.
  • Why. The generator needs a bounded, predictable context budget → a fixed k. The optional floor drops weakly-relevant filler after reranking, and is allowed to return [] on purpose, so the generator can say "no relevant context" rather than answer from junk — there is deliberately no min-1 fallback (that would defeat the floor).
  • Why NOT top-p. Top-p ("keep the smallest prefix whose cumulative probability mass ≥ p") only makes sense over a calibrated probability distribution that sums to 1 — e.g. a softmax over a vocabulary in token sampling. Cross-encoder / cosine relevance scores are not that: they don't sum to 1, aren't probabilities, and aren't even comparable across models (a "good" ms-marco-MiniLM score is a different number than a "good" bge-reranker score). So cumulative-mass thresholding is the wrong tool. The right tool is a fixed top-k plus an empirically-tuned- per-reranker absolute floor — which is exactly why the floor ships disabled by default (a hard-coded universal threshold would be meaningless across models).
  • Alternatives considered.
    • Fixed top-k only — simplest; the default behaviour.
    • Top-p / nucleus — inapplicable here (see above).
    • Dynamic-k by score gap — heuristic, brittle.
  • The tradeoff. The floor trades recall for precision on the tail; off by default keeps behaviour predictable, and the floor becomes opt-in once you've tuned it for your reranker.
  • When you'd choose differently. Precision-critical and you've measured a good threshold for your reranker → enable the floor. Recall-first → leave it off.
  • How it scales. Per-query-class k and floor; calibrate the floor on the golden set, per reranker.

LLM-as-judge for generation eval

  • Decision. A second, independent LLM grades each answer for faithfulness and answer-relevance against a strict JSON rubric.
  • Why. Retrieval being right doesn't make the answer right — a model can hallucinate a detail or answer a different question. You can't catch that by checking the pipeline ran; you must judge the answer's content against the exact context it was given. A structured verdict (scores + severities) lets you both gate one response and aggregate into a pass-rate regression. (Full treatment in Evaluation §3.)
  • Alternatives considered.
    • Human eval — gold standard; doesn't scale, can't run in CI.
    • n-gram metrics (ROUGE/BLEU) — need reference answers, measure surface overlap, blind to faithfulness.
    • NLI entailment models — cheaper, narrower.
    • No generation eval — the common gap: you'd measure retrieval and just hope the model used it faithfully.
  • The tradeoff. An LLM judge scales and is reproducible, but it's itself a model (cost per call; bias/variance) — mitigated by a strict structured rubric and a separable, unit-tested compute_gate. The design lets you pass a stronger model as judge than as generator.
  • When you'd choose differently. You have reference answers → add n-gram/exact-match as a cheap pre-filter. High-stakes → human eval on a sample + LLM judge on the rest.
  • How it scales. A held-out judge model, calibrated against human labels, with judge↔human agreement tracked over time; run as a CI eval gate on every change (roadmap).

Open-core plugin seam (RAGEVAL_PLUGINS_DIR / register_adapter / register_family)

  • Decision. Extend the engine with corpus-specific adapters + roster data through an import- time extension seam, never by forking. The public engine ships sample-only; a private overlay adds its corpus by file-presence.
  • Why an extension seam over a fork. The public/private boundary should be a file-presence boundary, not a code-diff. Point RAGEVAL_PLUGINS_DIR at a directory of adapter modules; the engine imports each at startup and each self-registers via the public APIs — register_adapter("<folder>", Adapter) maps a source-set folder to its adapter, and (optionally) register_family("<family>", "<tsv_stem>") maps a family to its roster TSV. An unset/absent dir is a clean no-op (engine stays sample-only); a present-but-broken plugin raises with the offending file path (a broken plugin is never silently skipped). The in-tree sample adapters keep working unchanged — this is purely additive.
  • Alternatives considered.
    • Fork the repo — private divergence → every upstream change is a merge conflict; the public/private split rots.
    • setuptools entry-points — heavier than warranted for an in-repo overlay.
    • A config flag listing adapters — still requires the adapter code in-tree (no clean separation).
  • The tradeoff. One import-time convention (a directory of *.py that call two public functions) buys a clean open-core split with zero core edits — the public engine drops the private adapters simply by their absence.
  • When you'd choose differently. A true plugin ecosystem with third-party distribution → setuptools entry-points or a published plugin-registry package. A single private overlay you control → this directory convention (lighter, in-repo).
  • How it scales. The same seam, with packaged adapters per source system and a governed registry — the open-core pattern carries straight to a real multi-corpus deployment.

How much complexity is warranted?

A teaching engine earns trust by being honest about where it stops. Strata-RAG deliberately favours readability and measurability over raw scale — and naming what it does not do is part of the design, because knowing what you didn't build, and why is the judgement an interviewer (or your future self) is actually looking for.

What this demo deliberately does NOT do:

  • No managed/distributed ANN — single-node Qdrant via docker compose.
  • No Qdrant native sparse vectors — BM25 is rebuilt in-memory at startup (fine for a medium corpus).
  • No multi-tenant access control — isolation is by collection name (enough to stop the sample/real contamination bug, not a tenancy model).
  • No token/structure-aware chunking — a pure character window.
  • No dashboards/tracing — flushed progress logs + a dry-run manifest + inspect (no Prometheus/ Grafana, no Langfuse/Phoenix).
  • No CI eval gate — the eval harness exists and is run by hand, not wired to block merges.
  • No multilingual stack — monolingual embedder/reranker (a known boundary, Engineering-Notes #3).
  • No agentic query router — the two-index design is built and proven; the router that dispatches between them is roadmap.

What an enterprise deployment adds (the seam each of the above leaves):

  • Managed ANN — Qdrant Cloud / cluster, quantization, on-disk payloads.
  • Native sparse retrieval — Qdrant sparse vectors, no startup rebuild.
  • Multi-tenant isolation — per-tenant collections / payload partitioning + access control.
  • Observability — Prometheus + Grafana (latency p50/p95, retrieval-score & eval distributions, token/cost, redaction counts), plus LLM-native tracing (Langfuse/Phoenix). Roadmap.
  • CI eval gates — retrieval + faithfulness eval blocking regressions on every change. Roadmap.
  • Multilingual retrieval — a multilingual embedder + reranker, proven on a non-EN golden slice. Roadmap (Phase-3 A/B).
  • Agentic routing — an LLM that reads a question and picks semantic vs aggregation vs both. Roadmap (Phase 2).

The roadmap items above are genuinely future work, not implemented here — the milestones track them. A demo that pretends to be production teaches nothing; a demo that says "here's exactly the seam where production picks up" teaches the shape of the real system, which is the whole point of this page.

Clone this wiki locally