Should I use RRF or weighted fusion? The notebook says RRF is the default but then measures it losing.

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

Notebook 04 says "default to RRF" and then measures RRF losing to BM25 alone. Which is the
advice?

What I have already tried

Read §4.9 twice. I think I follow the mechanism but I do not know what to actually do on a
new corpus.

Where

Notebook 04 §4.9

[akash-coded]

Both statements are right, and the resolution is a procedure rather than a preference.

The advice is: default to RRF, then measure.

RRF is the right default because it needs no labelled data, cannot be overfitted, and survives
score drift — a retriever whose score distribution shifts does not break the merge. If you have
no eval set, use RRF and do not think about it.

Once you have a labelled set large enough to tune α, measure. That is exactly what §4.9
does, and on this corpus the measurement overrode the default. The mechanism: RRF gives both
legs the same vote, and our dense leg is materially weaker than the lexical one here. An equal
vote drags the merge toward the weaker retriever.

So the procedure on a new corpus:

1. Start with RRF. Ship something.
2. Build the eval set (notebook 02 — this is the real work).
3. Sweep α. Compare against RRF and against each single leg — the second comparison is the
   one people skip, and it is the one that caught this.
4. If weighted wins, adopt it and put a re-tune on a schedule, because an α tuned on 200
   examples will not survive a corpus refresh.

What you must not do is copy α=0.2 from this repo. It is fitted to this corpus and this
encoder. With a modern encoder the dense leg gets stronger and α moves up — probably a lot.

One diagnostic worth running before you tune anything: slice evidence recall by query class
for each leg separately (§4.9 has the table). If your legs fail in the same places, fusion has
little to offer and you should fix the weaker leg instead. If they fail in different places —
which is the normal case — the size of the disagreement tells you roughly how much fusion can
buy you.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Marcus · cohort peer

Follow-up on the RRF side of this. k=60 looks like a magic number — it is in every RRF implementation I have seen and nobody explains it. If we are tuning α anyway, should we not be tuning k too? It seems odd to accept one hard-coded constant while carefully fitting another.

[akash-coded]

Dana · reviewer · owns the eval harness

Right to be suspicious of an unexplained constant, but the two knobs are not comparable in importance, and it is worth seeing why.

What k does. RRF scores a document as Σ_r 1/(k + rank_r), summed over retrievers. k is a damping constant that flattens the gap between top ranks. At k=0 the scores are 1, 1/2, 1/3… — rank 1 is worth twice rank 2, and a single retriever's top hit can dominate the merge. At k=60 they are 1/61, 1/62, 1/63… — nearly equal, so what matters is how many retrievers ranked the document at all, not how highly any one of them did.

That is the design intent: RRF is deliberately a voting scheme, not a scoring scheme. k=60 is from Cormack et al. (2009), chosen empirically on TREC data, and it is large precisely to suppress the influence of any one retriever's confidence.

Measured on our corpus, sweeping it:

 k     evidence_recall
  0          0.781
 10          0.796
 30          0.802
 60          0.804
120          0.803
300          0.799

Nearly flat from 30 to 120. The whole range is worth about +0.023, and everything above k=10 is within noise of everything else.

Compare that to α, which moves the same metric by +0.061 across its range on this corpus. One of these is worth your afternoon and it is not k.

The rule of thumb: tune k only if you have many retrievers with wildly different reliability, where you want to suppress a bad one's confident wrong answers. With two legs it is not where the signal is.

[akash-coded]

Marcus · cohort peer

The k=0 versus k=60 framing makes it click — it is a confidence-suppressor, not a free parameter. And +0.023 across the whole sweep versus +0.061 for α settles where to spend the time.

[akash-coded]

Priya · cohort peer

Can I ask the more basic question underneath this? Why 1/(k + rank) specifically? Why not 1/log(rank+1) like nDCG's discount, or just 1/rank? It looks arbitrary and I have never seen it justified, only used.

I would like to be able to answer this if I am asked it, rather than saying "it is what everyone uses".

[akash-coded]

Good question, and it is a real interview question — I have been asked it twice. There are three properties doing the work, and the form follows from wanting all three at once.

1 · It must consume ranks, not scores. BM25 returns values around 8–25; cosine returns values in [-1, 1]. There is no principled way to add those. Any monotone transform of a retriever's scores leaves its ranking unchanged, so the ranking is the only part that is comparable across legs. This kills every score-based fusion unless you first calibrate the scores — which needs labelled data, which is the thing you do not have when you reach for RRF.

2 · It must be steeply decreasing but bounded. Rank 1 should beat rank 50, but not by so much that one retriever's top hit wins outright. 1/(k+r) with a large k is nearly flat over the region you care about, so a document at rank 3 in both legs beats a document at rank 1 in one and rank 200 in the other. That is the behaviour you actually want from a merge and it is the entire argument for RRF.

3 · It must not require training. No parameter fitted to labels means nothing to overfit and nothing to re-tune when the corpus drifts.

On the alternatives you named:

• 1/rank satisfies 1 and 3 but fails 2 — rank 1 is worth twice rank 2, ten times rank 10. One confident retriever dominates. This is the failure k exists to prevent.
• 1/log(r+1) is the nDCG discount, and it is defensible. It decays more slowly in the tail than 1/(k+r) with large k. Empirically it performs similarly; RRF won on adoption, not on a proof.

The framing worth carrying into an interview: RRF is a positional voting system — closer to Borda count than to a scoring rule. Borda gives a candidate n - rank points; RRF gives 1/(k + rank), which is Borda with a convex weighting that cares more about the top of the list. Once you say that out loud, "why not a scoring rule" answers itself: scoring rules require interpersonal comparison of utility, and two retrievers' scores are not interpersonally comparable. Arrow's framing, borrowed for search.

The honest bit at the end: k=60 has no derivation. It is an empirical constant from Cormack et al. (2009) that turned out to be robust. If someone asks you to justify 60 specifically, the correct answer is that it was fitted on TREC and the result is insensitive across roughly 30–120 — which Dana's sweep above confirms on our corpus too.

[akash-coded]

Priya · cohort peer

The Borda framing is exactly what I was missing, and "two retrievers' scores are not interpersonally comparable" is the sentence I will actually remember.