rbpbound

ranx computes Rank-Biased Precision by multiplying each examined document by its raw graded relevance score, while the metric's own docstring defines the relevance term as binary. On judgments with graded relevance, the reported RBP is inflated by roughly the mean grade and can exceed 1, which no correct Rank-Biased Precision can reach.

The bug

ranx/metrics/rank_biased_precision.py documents (lines 58, 64):

RBP = (1 - p) * sum_{i=1}^{d} r_i * p^{i-1}
r_i is either 0 or 1, whether the i-th ranked document is non-relevant or relevant.

This is the canonical Moffat and Zobel (2008) definition, an expected per-document utility bounded in [0, 1]. The code (lines 28-35) uses the raw grade instead:

if run[i, 0] == qrels[j, 0]:
    weighted_hit_list[i] = qrels[j, 1]   # raw graded relevance, not 0/1
...
return (1 - p) * sum(weighted_hit_list * p ** np.arange(len(weighted_hit_list)))

clean_qrels only filters qrels[:, 1] >= rel_lvl; it never binarizes, so grades 2, 3, ... reach line 29 unchanged. The code is the wrong side, three ways: it contradicts its own docstring's r_i in {0, 1}, it violates the RBP [0, 1] bound, and every sibling metric in the package treats relevance as binary (_hits increments by 1.0, and precision and recall build on it).

What it does

On graded qrels the value is inflated, and a run of high-grade documents exceeds 1:

graded qrels: documented RBP = 0.360000, shipped RBP = 0.920000, inflation = 2.56x
six grade-3 documents: shipped RBP = 2.213568  (> 1 breaks the RBP bound)
binarizing the grades: shipped 0.920000 -> 0.360000, matching the documented 0.360000

The last line isolates the cause: replacing the grades with 1 recovers the documented value. Graded judgments are the norm in information retrieval (TREC relevance is graded 0 to 3, and nDCG in the same library requires the grades), so any evaluation on a graded collection gets a silently inflated, out-of-bounds RBP.

Layout

• excerpt.py: the documented binary formula and the graded code line, quoted with line numbers (MIT).
• rbp.py: RBP computed both ways, documented (binary) and shipped (graded), from the same formula.
• consequence.py: the graded inflation, the value above 1, and the recovery under binarization.
• test_rbpbound.py: documented and shipped disagree on grades, the inflation and the 2.21 value match the real library, the binary form stays within the bound, and binarizing recovers it.

Reproduce

python rbp.py
python consequence.py
python test_rbpbound.py

The values match ranx.evaluate(qrels, run, "rbp.8") on the current release; the fix is line 29, weighted_hit_list[i] = 1.0, matching the docstring and the sibling _hits.