Inference time when calulating MRR

[TamSiuhin]

Hi, @apoorvumang, this work is so great!

I try to reproduce the reported MRR, MR and H@N results on Wikidata5M with evaluate.py, but I found it needs quite a long time. Could you please provide an estimation of the evaluation time for calculating MRR?

Thanks!

[apoorvumang]

Hi, thanks for your interest.

evaluate.py does evaluation in the 'traditional' sense - it calculates log probabilities of all entity sequences (5M) using the Transformer model and then ranks these entities. It does not generate predictions, hence the long time.

Please use the code in eval_accelerate.py for the sampling generation technique (which is described in the paper as well). evaluate.py is not the code used for the numbers reported in the paper, eval_accelerate.py is used. Please see the Readme for more details. I would be happy to clarify any other doubts as well.

[TamSiuhin]

Thanks for your reply! But it seems that eval_accelerate.py just gives the unfiltered Hit@1, could you provide the official implementation for calculating MRR and filtered H@N? Appreciate your help!

[apoorvumang]

For that I would suggest you try the following:

1. Switch to branch apoorv-dump instead of main. This contains the latest code but is not cleaned, hence the large number of files.
2. In this branch, use eval_accelerate.py to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file.
3. Use notebook filtered_eval_from_scores.ipynb which uses these saved scores to evaluate MRR etc. I would suggest you go through the notebook to see the exact way these are calculated (if you see any issues please do contact me)

I will be cleaning the code/adding documentation etc. for this soon. Meanwhile I suggest you try the scores + notebook approach. Let me know if this works.

[TamSiuhin]

It seems to work great! Thank you so much for your patience!