Retrieval Pipeline

Retrieval Pipeline

Everything that happens before The-Trust-Layer gets to judge. Source: recall/retriever.py.

query → embed → ┬→ dense          (pgvector cosine)     ─┐
                ├→ sparse         (Postgres full-text)  ─┼→ RRF → rerank → top-k
                └→ learned sparse (SPLADE, optional)    ─┘

The two legs

Dense is pgvector cosine similarity over the query embedding. It finds memories that mean something similar, including ones sharing no vocabulary with the query.

Sparse is Postgres full-text search. It finds memories containing the query's actual terms — identifiers, error codes, proper nouns, anything an embedder tends to smear into a general neighbourhood.

Learned sparse is the optional SPLADE leg. It keeps sparse retrieval's inspectable term-like shape, but obtains weights from a transformer encoder rather than from Postgres lexical matching. It is a free local money path, but not a free latency path; on CPU it can dominate query time. Its most complete external measurement is the MTRAG Task A ladder in docs/MTRAG_BENCHMARK.md.

They fail differently, which is the entire argument for running both. Dense misses an exact token it has no strong representation for; sparse misses a paraphrase. Neither failure is rare in a memory corpus, where memos are written by one author across months and the query is written by someone (or something) recalling them imprecisely.

Each leg contributes a bounded candidate pool before fusion. The sparse legs can be disabled for ablation runs, which is how the eval harness isolates each contribution.

Fusion: Reciprocal Rank Fusion

RRF merges the two rankings by rank, not by score — each candidate accrues a contribution that decreases with its position in each list it appears in, and the totals are sorted.

Using ranks matters. Dense cosines and full-text relevance scores are not on a comparable scale and no fixed weighting makes them so; any attempt to blend the raw numbers is really a hidden hyperparameter that has to be re-tuned per embedder and per corpus. Ranks sidestep that entirely. A damping constant softens the weight of the very top positions so that neither leg can dominate on its own.

One detail with a consequence: every hit's reported score is its true dense cosine, including hits that arrived via the sparse leg — the sparse query computes the cosine for what it returns. So the score a caller sees is always the same quantity, comparable across hits, and directly usable by the calibrated threshold. A fused RRF score would not be: it is a ranking artefact with no absolute meaning, and thresholding it would be meaningless.

Reranking

An optional cross-encoder re-scores the candidates by reading the query and each candidate together, rather than comparing two independently-computed vectors.

The pool is reranked whole, then truncated to k — not truncated first. Slicing to k before reranking would hide a relevant document sitting just below the fused cutoff from the cross-encoder, and that document is exactly what reranking exists to rescue. Doing it in the wrong order produces a reranker that appears to work and cannot help in the only case that matters.

How much it buys, and when it buys nothing

On the standard public benchmark, with the default local embedder, reranking produced the largest single retrieval gain measured anywhere in this project — larger than the best embedder swap, with intervals disjoint from the baseline. Figures and arms → RESULTS §11 and FINDINGS §11.

An earlier version of this page said the opposite — that reranking was "redundant on an easy corpus with a strong embedder". That came from an early ablation on a small internal corpus and did not survive the benchmark above. Corrected rather than deleted, because it was published advice.

Two things still bound it. It is expensive, being a model pass over the whole candidate pool. And it only reorders what was retrieved: where the right document was never in the pool — measurable by looking at recall far beyond the top-k — reordering buys nothing, because the ceiling is the representation, not the pipeline. See Evidence-Map.

There is also a configuration that makes it buy nothing by construction: a candidate pool no wider than k hands the cross-encoder exactly the list it is meant to reorder. The flag is set, the model loads, the latency is paid, and the result is identical to having no reranker — with nothing reporting it. → Embedders-and-Rerankers

Embedders

Embedder is a protocol — anything with dim, name and embed. Five implementations ship; choosing between them, with the trade-offs and the evidence for each → Embedders-and-Rerankers.

	Use for
hashing	Fully offline, no model download, deterministic. Tests, CI, and ablation baselines where a weak embedder is the point.
FastEmbed (default)	Local ONNX embedding, no API key. The default for real use.
sentence-transformers	Any local model, including one you fine-tuned yourself.
Voyage	Cloud embeddings. Adds an API dependency, latency, and data egress.
OpenAI-compatible	Any endpoint speaking the OpenAI embeddings API — a gateway, or your own inference server.

name is not cosmetic: it is part of the embedding cache key and the calibration is rejected if it was fitted for a different embedder. Both exist so that switching models cannot silently reuse vectors or a threshold from the previous one.

Shared infrastructure sits around all four: batching, retry with backoff on transient failures, and a dimension check so that a model returning an unexpected vector width fails at the boundary rather than deep inside a SQL type error.

Is the cloud embedder worth it? Measured: it is corpus-conditional, and the old one-corpus rule was restated. On the private jargon corpus the hosted embedder wins clearly; on one ordinary PEPs corpus the gap is inside the noise; across 17 held-out corpora the hosted embedder wins almost everywhere and the gap tracks corpus size more than unusual vocabulary. The rule is not "always buy the better embedder"; it is "benchmark your corpus before choosing." Evidence → FINDINGS §7–§8 and results/gap/FINDINGS-embedder-gap.md.

Fine-tuning follows the same conditional logic: it pays for a vocabulary gap and pays nothing on a corpus the base model already handles. Controlled study, including the null → docs/RAG_TRAINING_STUDY.md.

Indexing

Source: recall/index.py.

Chunking is structure-aware — prose splits on block boundaries with overlap, code has its own path — and oversized blocks are split rather than truncated.

Beyond the obvious, the indexer carries several guards that exist because of specific failures:

• Content-hash skip. Unchanged files are not re-embedded. This is what makes re-indexing a large corpus cheap enough to run daily.
• Bounded-memory batched writes, so a large corpus does not have to fit in RAM.
• NUL sanitisation. A single NUL byte in one file previously aborted an index run over hundreds of files. Postgres text cannot hold it; one stray byte should not cost the whole run.
• Index-root confinement, which survives symlinks — and is implemented without relying on a standard-library feature that only exists in newer Pythons than the project supports.
• Fail-fast on malformed validity dates, so bad metadata is refused at write time rather than becoming an invalid_metadata verdict at read time.
• Pruning of vanished sources, so files deleted from disk leave the index.
• The prune guard. A re-index that would delete a large fraction of the sources under a root raises rather than proceeding, and deletes nothing. This is how a missing corpus (wrong path, unmounted volume, failed checkout) stops being indistinguishable from a deleted one. It is a behaviour change for scripted re-indexing: confirm the files really are gone, then re-run with the override flag.

An embedding cache sits alongside, content-addressed by embedder name, dimension and text — so re-indexing after a restart, or across a corpus with repeated boilerplate, does not re-pay for vectors already computed.

Tuning knobs, and the honest note about them

Candidate pool size, chunk size, learned sparse retrieval and reranking are all adjustable. Before spending time on them, read Evidence-Map: on one real corpus, larger candidate pools and every chunk-size variant moved retrieval by nothing at all, while the embedder moved it substantially. On LOCOMO and MTRAG, reranking and SPLADE were real levers. The lesson is not a universal ranking of knobs; it is to measure whether your cap is representation, pool recall, ordering, or answerability.

The practical order of investigation that follows: measure whether the right document is being retrieved at any depth first. If it is not, no amount of reordering or pool-widening will help, and the representation is what needs to change.

ANN parameters are a separate matter, and the honest framing of them matters too. The filtered query path sets a wider search and relaxed iterative scan because an HNSW walk is filter-blind: it walks to the globally nearest neighbours and only then discards the ones failing the filter, so a selective filter exhausts the candidate list before k matches are found. The symptom is truncation — asking for ten results and silently getting three.

That is what the tuning fixes, and it is worth being precise that this is not the same as improving recall. Two measurements were taken; they agree that truncation is eliminated and disagree on recall, and both are published — on a fixture corpus rebuilt until it reproduces the pathology recall improves, while on a corpus built the way a real multi-file index run builds one it moves slightly the other way. relaxed_order fills to k with approximate matches, so the trade is truncation for approximation. Figures → FINDINGS.

The unfiltered path still runs at the defaults, where it measured well — but every query now also carries a tenant predicate, and that combination is not yet measured. Background: issue #11 and PR #57.

---

Next: The-Trust-Layer for what happens to these hits · Configuration-Reference for the ANN and indexing knobs · Evidence-Map for the measurements behind every claim here.