You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I understood that, when evaluating MAP@k, relevance judgment scores equal to 0 are ignored.
In my case, I get a bit of a weird behaviour.
I'm working on a balanced dataset with binary relevancy and define qrels by including both 1s and 0s documents.
While ndcg@10 gives me results at about 0.7, MAP@10 is extremely low at about 0.10.
Can this be because, besides the very first documents, the model perform poorly or am I doing something wrong when evaluating?
Almost all the metrics ignore qrels with zero scores, including NDCG.
So the difference you get is not because of that.
However, I think your results are entirely possible if you have many relevance judgments for each query.
Note that the Average Precision denominator is equal to the number of relevant documents, regardless of the cut-off.
Conversely, DCG and, therefore, NDCG depends on the cut-off and not on the number of relevant documents. (Please, cross check this for correctness.)
I understood that, when evaluating MAP@k, relevance judgment scores equal to 0 are ignored.
In my case, I get a bit of a weird behaviour.
I'm working on a balanced dataset with binary relevancy and define qrels by including both 1s and 0s documents.
While ndcg@10 gives me results at about 0.7, MAP@10 is extremely low at about 0.10.
Can this be because, besides the very first documents, the model perform poorly or am I doing something wrong when evaluating?
predictions in test_loaded_pdf is not a list of binary relevancy but it's a float relevancy score
The text was updated successfully, but these errors were encountered: