Architecture

Architecture

The core design insight: two classes of question

Real questions over a document corpus split into two classes, and a pure embedding-RAG fails the second:

Example question	Class	Mechanism
"which projects use a citrus theme?"	semantic retrieval	embed → top-k similarity
"list every owner and count projects per owner"	aggregation	structured metadata → GROUP BY + COUNT
"themes used in both source-sets"	set intersection over a facet	metadata filter + semantic

A vector top-k cannot count, and it cannot compute a true set intersection — those are exact operations over structured fields, not nearest-neighbour search. So ingest produces both a semantic index (Qdrant) and a structured metadata table (SQLite sidecar). Phase 1 builds both and the eval to prove retrieval; Phase 2 adds an agentic router that picks the right mechanism per question.

This is the single decision that shapes everything downstream: enrichment exists to populate the sidecar, the sidecar exists to answer aggregation, the eval is split by question kind, and the future router exists to dispatch between the two.

The pipeline

corpus (source-set / project / documents)
   │  sources/         ── ADAPTERS discover heterogeneous docs (.md/.txt/.docx) → SourceDoc
   ▼
candidate SourceDocs
   │  classify        ── TIER-1 RULES decide INCLUDE/EXCLUDE per corpus intent
   │  manifest        ── DRY-RUN: include/exclude-with-reason + coverage (blind spots) — no embedding
   ▼
included docs
   │  redact          ── scrub secret VALUES + policy-aware PII before anything downstream sees them
   │  chunking        ── split into overlapping windows
   │  embeddings      ── chunk text → vectors (local sentence-transformers)
   │  index           ── upsert vectors + payload into QDRANT (HNSW)
   │  enrich          ── LLM → structured metadata (name / category / theme tags / summary)
   │  sidecar         ── persist structured records to SQLITE (exact aggregation)
   ▼
indexed engine
   │  retrieve        ── HYBRID dense+BM25 → RRF FUSION → CROSS-ENCODER RERANK → top-k
   │  generate        ── augmented (guarded) prompt → grounded answer + citations
   │  eval            ── Recall@K / Precision@K / MRR / nDCG (golden set) + LLM-judge faithfulness
   │  inspect         ── browse the actual chunks / coverage / sidecar / quality flags
   ▼
{answer, sources, eval}   ←── served by the API (POST /ask)

Stage notes

Source adapters. The only place that knows a corpus's filesystem layout. Each yields normalised SourceDocs, so the rest of the pipeline is corpus-agnostic — a new corpus shape is a new adapter class plus one registration call, and nothing else changes. The engine never copies the corpus into the repo; custom indexes and sidecars are gitignored.

Tiered relevance classification. "Noise vs signal" is relative to a stated corpus intent. Deterministic rules (Tier 1, the trusted committed artifact) do the overwhelming majority of the work; an LLM advisor (Tier 2) only proposes rule changes a human approves. The untrusted model proposes; the committed ruleset enforces.

Dry-run manifest. Embedding is the last step — the pipeline must be inspectable before it. The manifest shows what would be included/excluded (with the deciding rule) plus a coverage report (blind spots and outliers) and the redaction count, all before a single vector is built.

Redaction. Secret values are scrubbed after extraction and before chunking, for every included document, so no downstream stage (embed, payload, enrich) ever sees a credential — a defense-in-depth layer alongside the exclusion rules. PII detection is a pluggable detector behind a fixed keep-or-redact policy.

Vector index (Qdrant + HNSW). The ANN graph Qdrant searches. Build-time (M, ef_construct) and query-time (ef_search) knobs trade recall against speed and RAM. The collection name is derived from the embedding model, so two models' indexes coexist and can be A/B-compared like with like; the synthetic sample gets an isolated collection so it can never upsert into a custom-corpus index.

Hybrid retrieval → RRF → rerank. Dense vectors capture meaning; sparse BM25 captures exact terms — each covers the other's blind spot. Their two ranked lists have incomparable scores, so Reciprocal Rank Fusion combines them using rank position alone (1/(k+rank) summed); items ranked high by both rise. A slow-but-precise cross-encoder then re-scores only the fused candidates, scoring (query, chunk) together — retrieve-then-rerank gives corpus-scale recall with pair-scale precision. An optional rerank-score floor can drop weak filler so the generator can refuse rather than answer from junk.

Metadata sidecar. A SQLite table for the exact aggregation/intersection a vector top-k can't do, and an audit surface (a NULL extracted field flags an enrichment failure).

Generation. An augmented prompt produces a grounded, cited answer. The prompt is hardened with layered prompt-injection defenses (input scan, random-sentinel spotlighting, instruction hierarchy, output validation), and every response carries a guardrail report — silent defenses can't be trusted.

The two-query-class design, restated

The architecture is best understood as two indexes serving two question classes from one ingest:

• The semantic index answers "what is this about / which are similar?" — fuzzy, meaning- based, ranked.
• The metadata sidecar answers "how many / which set / which exact value?" — precise, structured, exact.

Phase 1 builds both and proves each with its own eval slice. Phase 2's query router is simply the component that reads a question and decides which of these two mechanisms (or both, for a filtered-then-semantic hybrid) should answer it.