Questions about MRR on WikiKG90Mv2

[YueFan1014]

Hello, I am interested in the evaluation on WikiKG90Mv2. In the task description:

'The model is asked to predict the top 10 tail entities that are most likely to be positive. The goal is to rank the ground-truth tail entity as high in the rank as possible within the top 10, which is measured by Mean Reciprocal Rank (MRR).'

I am wondering how many ground-truth tail entities are calculated for each (h, r) entry? If only the most ranked-ahead tail is calculated for each (h, r), does it mean that for this (h, r) it only has one tail? Thanks!

[weihua916]

Hi, this is a great question. Yes, that's our assumption. It is very likely that (h, r) only has a single tail t_test in the test set. Note that you should not predict the training tail t_train for the given (h, r), even though (h, r, t_train) is a factually correct triple.

[YueFan1014]

OK, thanks!

[YueFan1014]

Hello! We found that in the validation, test-dev and test-challenge sets, some queries (head, relation) appear over 10 times in the same set. Does it mean that for a query (h_1, r_1) with 15 true tails (t_1, t_2, ..., t_15) in the test set, we have to predict the 15 tails accordingly to their hidden permutation in the test set in order to achieve a higher MRR score? For example, for the 1st (h_1, r_1), we put t_1 in the first, and for the 2nd (h_1, r_1), we put t_2 in the first. Besides, if we only recall top-10 tails for (h_1, r_1), since it appears 15 times in the test set, Does it mean that at least 5 queries of (h_1, r_1) achieve 0 scores? Hoping for your reply!

[weihua916]

This is a great question!
Indeed, there is (h, r) appearing 15 times on the test-challenge set, which means that there are 15 correct tail entities for the given (h, r). What you said is correct; our evaluation protocol cannot handle this situation, but such a case is pretty rare, so I do not think it will affect the average metric much.

For repeated (h, r) data, I'd suggest you to let your model make the same top10 prediction without changing the prediction as you suggested. If your model is perfect, your model will receive scores of 1, 1/2, 1/3, ... for the repeated (h, r) entry. I think this is fine as this won't overemphasize the same (h, r) in our averaged metric. Of course, this also means that the highest possible MRR is no longer 1, but that's not a big problem as long as the metric can differentiate different methods. Hope this clarifies your concern.