You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MRR@k is only equal to Recall@k if you have exactly one positive per query. Unfortunately, there are many meanings for Recall, but at least for trec eval (which ranx follows) it always uses positives_found/total_positives, even if k is smaller than the amount of positives you have.
For the other definitions, one I have seen called as R_cap (using positives_found/min(k,total_positives)) or Success ( the function "found any positives @ k").
Recall@1 is equal to MRR@1 only if you have exactly 1 relevant document per query, otherwise they are not as Recall takes into consideration the total number of relevant documents (retrieved + non-retrieved).
You can find the definition of both metrics in ranx's documentation (Recall - MRR) and Wikipedia (Recall - MRR).
Describe the bug
MRR@1
should be equal toRecall@1
. However, these metrics diverge for the case below.To Reproduce
or use Google Colab.
Expected behavior
mrr@1
=recall@1
Am I missing something?
The text was updated successfully, but these errors were encountered: