Embedders and Rerankers

Embedders and Rerankers

The two settings that move retrieval quality most, and the one interaction between them that silently disables the more valuable of the two.

This page is about choosing. It states no figure and no default — those live with the code that produced them, and every claim below links to where. For the variable names and their current values see Configuration-Reference; for what to pip install see Installation-and-Setup.

What you are actually deciding

Retrieval runs in two stages, and the settings belong to different ones.

The embedder decides what can be found at all. It turns your text and your query into vectors, so a memory it represents poorly is not merely ranked low — it is absent from the candidate pool and no later stage can recover it.

The reranker decides the order of what was found. It reads query and candidate together instead of comparing two independently-computed vectors, which lets it judge relevance an embedder can only approximate. It cannot rescue a document the embedder never surfaced.

That asymmetry is the whole basis for choosing between them: if the right memory is not being retrieved at any depth, the embedder is your problem and the reranker cannot help. Measure which case you are in before spending on either — Evidence-Map links the study that made this a rule rather than a guess.

Embedders

Five implementations ship. All satisfy the same protocol, so switching is a configuration change, not a code change.

Hashing

Deterministic, no model, no download, no network. Its vectors carry no semantics — it is a deliberately weak baseline.

Pick it for tests, CI, and ablations where a weak embedder is the point: it is how the harness demonstrates that the sparse leg and the reranker are contributing something. Do not pick it for real retrieval. It is not a lightweight model; it is not a model.

FastEmbed — the default for real use

Local ONNX inference. No API key, no egress, runs offline, and the model downloads once.

Pick it unless you have measured a reason not to. It is the configuration nearly every published result here was produced on, which also makes it the one whose failure modes are best mapped. Costs you local CPU at index time and a first-run model download.

sentence-transformers

Any local model from the ecosystem, including one you fine-tuned yourself.

Pick it when you need a specific model FastEmbed does not package, or when you have a genuine vocabulary gap and have fine-tuned for it. That second case is conditional and was measured both ways: fine-tuning pays on a corpus of private jargon and pays nothing on prose a base model already handles — the controlled study, including its null result, is docs/RAG_TRAINING_STUDY.md. Costs you a heavier dependency tree than FastEmbed.

Voyage

Cloud embeddings from Voyage AI.

Pick it when your corpus benchmark says the hosted model is worth the trade. The old narrow rule, "only for idiosyncratic vocabulary", was restated after a larger held-out study: corpus size also predicts the gap, and the hosted embedder won on almost every held-out corpus measured. The cross-corpus comparison is results/gap/FINDINGS-embedder-gap.md and FINDINGS §7–§8, which publish the private corpus where it wins clearly, the PEPs corpus where the difference sits inside the noise, and the 17-corpus restatement. Costs you an API key, per-query latency, an external dependency in your retrieval path, and — the one that decides it for some readers — your queries and documents leave your machine. That is a change in posture, not just in latency.

OpenAI-compatible

Any endpoint speaking the OpenAI embeddings API, including gateways and self-hosted servers.

Pick it to reach a hosted model this project does not package directly, or to point at your own inference server. It is the path the third-party-benchmark arms use when a comparison requires matching someone else's embedder rather than choosing the best one. Costs you the same egress and dependency posture as any cloud embedder, minus the guarantee that the endpoint behaves — you own that.

Learned sparse retrieval

SPLADE is not an embedder replacement in the dense-vector sense. It is an optional learned sparse leg that enters fusion beside dense and lexical retrieval.

Pick it when lexical matching is too brittle but you still want a sparse signal, and when your latency budget can tolerate transformer sparse encoding. On MTRAG Task A, the learned sparse leg was a meaningful local lever and the best free-local retrieval arm used it. The current table is in docs/MTRAG_BENCHMARK.md.

Costs you model loading and CPU or GPU time. It is free in API spend, not free in query latency.

The coupling nobody expects

An embedder's name is part of its identity, not a label. It keys the embedding cache, and a calibration fitted for one embedder is rejected rather than reused for another.

Both exist because the alternative is silent corruption: vectors from the old model mixed with the new, or an abstention threshold tuned for a cosine distribution that no longer exists. Switching embedders means re-indexing and re-calibrating. The system will refuse the shortcut rather than serve you a plausible wrong answer — which is the same principle as everything in The-Trust-Layer.

Why a threshold cannot simply be carried across: each model's cosines live in a different regime, and this was measured across models and corpora rather than assumed. See Evidence-Map.

Rerankers

Two implementations: a no-op, and a cross-encoder.

The no-op is the default, and that default is about cost, not about value — a cross-encoder is a model pass over the whole candidate pool, and it is the most expensive thing in the query path.

The cross-encoder

Pick it if you care about retrieval quality and can afford the latency. On the standard benchmark it produced the largest single retrieval gain measured anywhere in this project, using the default local embedder — bigger than the best embedder swap, with intervals disjoint from the baseline. The measurement, its arms and its cost: RESULTS §11 and FINDINGS §11.

Costs you roughly a second of query latency and a model download. That is the trade, and for an interactive path it is a real one.

⚠️ An earlier version of this wiki called reranking "redundant on an easy corpus with a strong embedder". That framing came from an early ablation on a small internal corpus and did not survive the benchmark run above. It is corrected here rather than deleted, because it was published advice and someone may have acted on it.

Which cross-encoder model

Two were measured head to head. The larger, more modern one is indistinguishable from the shipped default while costing several times more per query — which is the interesting result, because it means the gain belongs to reranking as a stage, not to model capacity. The comparison is in RESULTS §11.

Practical consequence: do not reach for a bigger cross-encoder expecting more. It was tried and it bought nothing here.

A hosted Voyage reranker was measured separately on the answerability ladder and beat the shipped local reranker on that corpus. It is also an API dependency and must not be stacked blindly with a weaker local ranker. Current figures and caveats live in RESULTS §12 and FINDINGS §12.

The shipped model is pinned to an exact revision on the model hub, for the same reason the calibration is embedder-keyed — an unpinned reference is mutable, and a silently swapped model changes every result that depends on it.

⚠️ The interaction: a reranker that cannot do anything

A reranker can only reorder the candidates it is given. If the candidate pool is no wider than the number of results you ask for, the reranker receives exactly the list it is supposed to reorder, and reordering it changes nothing that survives truncation.

This is the failure that produces a configuration which looks correct — the flag is set, the model loads, the latency is paid — and measures identical to having no reranker at all. Nothing errors. It is the same shape as every other defect this project publishes: a plausible result with no signal that it is wrong.

So: widen the candidate pool when you enable reranking. The pool is a separate knob from k, and leaving it at a value tuned for un-reranked retrieval is what disables the stage. The pool also has an upper bound imposed by the ANN search width, so it cannot be raised without limit — the current values and that ceiling are in Configuration-Reference.

The pool is also reranked whole and then truncated, never truncated first. Slicing before reranking would hide the document sitting just below the fused cutoff from the cross-encoder, and that document is precisely what reranking exists to rescue. See Retrieval-Pipeline.

Choosing, in order

1. Find out whether the right memory is retrieved at any depth. If it is not, this whole page is about the wrong stage — change the representation. Evidence-Map.
2. Start on the local default. It is free, private, offline, and it is the configuration the published evidence was produced on.
3. Turn on the reranker if query latency allows. On the evidence here it is the largest single improvement available — and widen the candidate pool with it, or you have paid for nothing.
4. Only then consider a cloud embedder, and only with a corpus-specific benchmark. It changes your privacy posture, and the measured gain is conditional on the corpus.
5. Re-index and re-calibrate after any embedder change. Not optional; the system enforces it.

---

Next: Configuration-Reference for the variables and their current values · Retrieval-Pipeline for how these stages fit together · Evidence-Map for every measurement referenced above.