Faulty Scores Generated by Evaluator in Presence of Empty Relevance Sets

[DeepaliP98]

When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:

qrel = {
'q1': {
'd1': 0,
'd2': 1,
'd3': 0,
},
'q2': {
},
'q3': {
'd2': 1,
'd3': 1,
},
}

evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
evaluator.evaluate(some_valid_b)

In this case, the evaluator provides erroneous scores for 'q2' and relations after q2 i.e q3, without raising any warning, which could mislead users.