-
Notifications
You must be signed in to change notification settings - Fork 0
Engineering Notes
Four debugging stories from building Strata-RAG, told as teaching cases. Each is a bug that was surprising — it looked like one kind of problem and turned out to be another — and each left a durable design principle behind. The specifics are genericised; the lessons are the point.
A query against what should have been a clean index returned an answer citing a source nobody recognised. The cited passage was fixture text — content that only exists to exercise the test suite. It had no business being retrievable from a real query.
The stray citation was the tell. Tracing it back: a test run and a real ingest had written into the same collection. The vector index doesn't know or care where a chunk came from — once two corpora share a collection, every query searches the union of both. The test data wasn't "leaking" through some subtle channel; it was simply in the index, indistinguishable from real content at query time.
The fix was structural, not a patch: the collection name is derived from the corpus, and the synthetic sample gets its own suffixed collection that a real ingest can never write into. Isolation is enforced by naming, so it can't be forgotten — a sample ingest and a real ingest physically cannot land in the same place. And if the serving side is pointed at a different corpus than was ingested, the engine fails fast with a clear preflight error naming the expected collection, instead of silently searching the wrong (or an empty) index.
Principle: retrieval indexes have no built-in provenance. If two corpora can share a collection, eventually they will. Make isolation a property of the name, and make a mismatch a loud failure, not a silent wrong answer. A stray citation is a contamination alarm — listen to it.
The deterministic leak gate — a fast, dependency-free regex scan — was green. Yet a reviewer reading the same content could tell, from its shape, that it encoded something it shouldn't: not any single forbidden word, but the structure, the field arrangement, the overall pattern. No literal token a regex matches on; a meaning a human reads at a glance.
Regex matches tokens, not meaning. A leak that lives in the arrangement of otherwise- innocent words — the way fields line up, an implied context — has no fixed string to grep for. Tightening the regex was a losing game: every pattern added was either too narrow (missed the next variant) or too broad (flagged clean text). The structural signal simply isn't expressible as a token match.
The answer was not a better regex but a second, different kind of gate. The deterministic scan stays as the hard gate: cheap, offline, zero false-negatives on the concrete tells it can see (real paths, key/token assignments, private-key blocks). On top of it sits a semantic gate — a hostile-reviewer model that reads the changed files and asks "would this leak private context or look bad to publish?", catching the shape and ambiguous cases a regex structurally cannot. The two compose: deterministic catches what's certain and fast; semantic catches what needs judgement. Neither alone is sufficient.
Principle: some leaks are semantic, not lexical. A regex defends the literal; a model defends the implied. Run both — the deterministic one as the non-negotiable hard gate, the semantic one as the judgement layer — and never assume green on one means clean.
A query in one language returned nothing useful, even though the corpus plainly contained relevant material — in another language. The retrieval was not "low quality"; it was near- empty, as if the content didn't exist. Yet the same content was trivially retrievable by an equivalent query in its own language.
The embedder and the cross-encoder reranker were monolingual. A monolingual model maps text from a language it wasn't trained on into a different, essentially unrelated region of the vector space — so a query embedding and its true answer's embedding end up far apart purely because they're in different languages. Cosine similarity then does exactly what it's told and ranks the genuinely relevant chunk near the bottom. Nothing was broken; the model was answering a question about the wrong space. The reranker, also monolingual, couldn't rescue it either.
Cross-lingual retrieval is not a tuning knob on top of an existing stack; it's a property of the models you chose. If retrieval must cross languages, the embedder and reranker must be multilingual — and you must prove it on a golden set that actually contains non-English questions mapped to (possibly English) answers. Hence the Phase 3 A/B: monolingual vs multilingual stack, evaluated on EN and non-EN slices, with any English regression measured. You cannot assert cross-lingual capability; you measure it, because the model defines the boundary.
Principle: retrieval can only cross a language boundary the embedding model was built to cross. A monolingual embedder silently caps recall at its own language — the failure looks like "no results," not "wrong model." Choose the model for the boundary, then prove it with a bilingual eval.
An enrichment pass asked the LLM to extract an authoritative attribute for each item — a label that identifies who the item belongs to. The model returned a confident value for every item. Spot-checking, the values were wrong in a meaningful fraction of cases: plausible, fluent, and incorrect.
The attribute simply wasn't present in the documents. It's an externally-assigned label — the kind of fact a project's own prose has no reason to state. Asked to extract something that isn't there, the LLM did what LLMs do: it inferred a plausible answer from surrounding context and reported it with full confidence. There was no hallucination detector failure here — the eval for faithfulness was about grounding in retrieved text, and this value wasn't in the text to be grounded against. The model was confidently filling a gap that no amount of prompt-tuning could close, because the information lives outside the corpus.
When an attribute is authoritative and external, the correct source is an external record of record — a roster / mapping table keyed by item id — not LLM inference over the docs. So that attribute is populated by a deterministic join against the authoritative table, and a reconciliation step reports MATCH / MISMATCH / missing between the table and whatever the docs do say. Mismatches are flagged for human adjudication, not auto-resolved, and the trust order is explicit (authoritative record > doc-stated > inferred). The lookup is, by its nature, a structured-metadata query (a SQL join), not semantic retrieval — and recognising that is itself the finding.
Principle: an LLM asked for a fact that isn't in its context will confidently invent a plausible one. For attributes that are authoritative and externally owned, use a deterministic ground-truth join and a reconciliation/audit step — never LLM inference. Confidence is not evidence.
Each story is the same shape: a failure that looked like one thing (a quality issue, a missed keyword, a bad answer) was actually a boundary being crossed silently — a corpus boundary, a lexical-vs-semantic boundary, a language boundary, a what's-in-the-documents boundary. The durable fixes all make the boundary explicit and enforced: isolate by name, layer the defenses, choose the model for the boundary, join to ground truth. The bug is rarely where the symptom is.