Embedders and Rerankers

Embedders and Rerankers

The two settings that move retrieval quality most, and the one interaction between them that silently disables the more valuable of the two.

This page is about choosing. It states no figure and no default — those live with the code that produced them, and every claim below links to where. For the variable names and their current values see Configuration-Reference; for what to pip install see Installation-and-Setup.

What you are actually deciding

Retrieval runs in two stages, and the settings belong to different ones.

The embedder decides what can be found at all. It turns your text and your query into vectors, so a memory it represents poorly is not merely ranked low — it is absent from the candidate pool and no later stage can recover it.

The reranker decides the order of what was found. It reads query and candidate together instead of comparing two independently-computed vectors, which lets it judge relevance an embedder can only approximate. It cannot rescue a document the embedder never surfaced.

That asymmetry is the whole basis for choosing between them: if the right memory is not being retrieved at any depth, the embedder is your problem and the reranker cannot help. Measure which case you are in before spending on either — Evidence-Map links the study that made this a rule rather than a guess.

Embedders

Five implementations ship. All satisfy the same protocol, so switching is a configuration change, not a code change.

Hashing

Deterministic, no model, no download, no network. Its vectors carry no semantics — it is a deliberately weak baseline.

Pick it for tests, CI, and ablations where a weak embedder is the point: it is how the harness demonstrates that the sparse leg and the reranker are contributing something. Do not pick it for real retrieval. It is not a lightweight model; it is not a model.

FastEmbed — the default for real use

Local ONNX inference. No API key, no egress, runs offline, and the model downloads once.

Pick it unless you have measured a reason not to. It is the configuration nearly every published result here was produced on, which also makes it the one whose failure modes are best mapped. Costs you local CPU at index time and a first-run model download.

sentence-transformers

Any local model from the ecosystem, including one you fine-tuned yourself.

Pick it when you need a specific model FastEmbed does not package, or when you have a genuine vocabulary gap and have fine-tuned for it. That second case is conditional and was measured both ways: fine-tuning pays on a corpus of private jargon and pays nothing on prose a base model already handles — the controlled study, including its null result, is docs/RAG_TRAINING_STUDY.md. Costs you a heavier dependency tree than FastEmbed.

Voyage

Cloud embeddings from Voyage AI.

Pick it when your corpus vocabulary is idiosyncratic enough that a local model smears it. That is a measurable condition, not a preference: the cross-corpus comparison is results/gap/FINDINGS-embedder-gap.md and FINDINGS §7–§8, which publish both the corpus where it wins clearly and the corpus where the difference sits inside the noise. Costs you an API key, per-query latency, an external dependency in your retrieval path, and — the one that decides it for some readers — your queries and documents leave your machine. That is a change in posture, not just in latency.

OpenAI-compatible

Any endpoint speaking the OpenAI embeddings API, including gateways and self-hosted servers.

Pick it to reach a hosted model this project does not package directly, or to point at your own inference server. It is the path the third-party-benchmark arms use when a comparison requires matching someone else's embedder rather than choosing the best one. Costs you the same egress and dependency posture as any cloud embedder, minus the guarantee that the endpoint behaves — you own that.

The coupling nobody expects

An embedder's name is part of its identity, not a label. It keys the embedding cache, and a calibration fitted for one embedder is rejected rather than reused for another.

Both exist because the alternative is silent corruption: vectors from the old model mixed with the new, or an abstention threshold tuned for a cosine distribution that no longer exists. Switching embedders means re-indexing and re-calibrating. The system will refuse the shortcut rather than serve you a plausible wrong answer — which is the same principle as everything in The-Trust-Layer.

Why a threshold cannot simply be carried across: each model's cosines live in a different regime, and this was measured across models and corpora rather than assumed. See Evidence-Map.

Rerankers

Two implementations: a no-op, and a cross-encoder.

The no-op is the default, and that default is about cost, not about value — a cross-encoder is a model pass over the whole candidate pool, and it is the most expensive thing in the query path.

The cross-encoder

Pick it if you care about retrieval quality and can afford the latency. On the standard benchmark it produced the largest single retrieval gain measured anywhere in this project, using the default local embedder — bigger than the best embedder swap, with intervals disjoint from the baseline. The measurement, its arms and its cost: RESULTS §11 and FINDINGS §11.

Costs you roughly a second of query latency and a model download. That is the trade, and for an interactive path it is a real one.

⚠️ An earlier version of this wiki called reranking "redundant on an easy corpus with a strong embedder". That framing came from an early ablation on a small internal corpus and did not survive the benchmark run above. It is corrected here rather than deleted, because it was published advice and someone may have acted on it.

Which cross-encoder model

Two were measured head to head. The larger, more modern one is indistinguishable from the shipped default while costing several times more per query — which is the interesting result, because it means the gain belongs to reranking as a stage, not to model capacity. The comparison is in RESULTS §11.

Practical consequence: do not reach for a bigger cross-encoder expecting more. It was tried and it bought nothing here.

The shipped model is pinned to an exact revision on the model hub, for the same reason the calibration is embedder-keyed — an unpinned reference is mutable, and a silently swapped model changes every result that depends on it.

⚠️ The interaction: a reranker that cannot do anything

A reranker can only reorder the candidates it is given. If the candidate pool is no wider than the number of results you ask for, the reranker receives exactly the list it is supposed to reorder, and reordering it changes nothing that survives truncation.

This is the failure that produces a configuration which looks correct — the flag is set, the model loads, the latency is paid — and measures identical to having no reranker at all. Nothing errors. It is the same shape as every other defect this project publishes: a plausible result with no signal that it is wrong.

So: widen the candidate pool when you enable reranking. The pool is a separate knob from k, and leaving it at a value tuned for un-reranked retrieval is what disables the stage. The pool also has an upper bound imposed by the ANN search width, so it cannot be raised without limit — the current values and that ceiling are in Configuration-Reference.

The pool is also reranked whole and then truncated, never truncated first. Slicing before reranking would hide the document sitting just below the fused cutoff from the cross-encoder, and that document is precisely what reranking exists to rescue. See Retrieval-Pipeline.

Choosing, in order

1. Find out whether the right memory is retrieved at any depth. If it is not, this whole page is about the wrong stage — change the representation. Evidence-Map.
2. Start on the local default. It is free, private, offline, and it is the configuration the published evidence was produced on.
3. Turn on the reranker if query latency allows. On the evidence here it is the largest single improvement available — and widen the candidate pool with it, or you have paid for nothing.
4. Only then consider a cloud embedder, and only if you have measured a vocabulary gap. It is the option that changes your privacy posture, and it is conditional on the corpus.
5. Re-index and re-calibrate after any embedder change. Not optional; the system enforces it.

---

Next: Configuration-Reference for the variables and their current values · Retrieval-Pipeline for how these stages fit together · Evidence-Map for every measurement referenced above.