The Evaluation of the index and model doesn't match of the reported paper

[Rachel-guo]

Hi all,

Thanks for this interesting work and making the code available. I am Buruo, an MSc student at the University of Glasgow, studying with @cmacdonald. I’m trying to make a PyTerrier plugin for JPQ.

I am using your pre-built indices, but my evaluation doesn't match those reported in the paper. My best NDCG@10 is 0.5568 on the TREC 2019 Deep Learning track Passage Task.

For us to debug our integration, would you be able to provide results files - i.e. rankings generated by JPQ. This would also be useful for others wishing to compare the JPQ results in their own papers.

Buruo

[jingtaozhan]

Hi, I'm very happy to help!
I wonder whether you could reproduce the results using the code from this repo and whether the results on msmarco dev set is expected.
I will rerun the experiments today to check the code :). I'll share the ranking files once the check is over.

Best,
Jingtao

[jingtaozhan]

Hi Buruo,

I can get the reported results using the code in this repo. I don't know why you couldn't. Maybe you can provide me more details so we can work it out.

I open source the ranking results per your request.

I also provide several useful scripts to help you reproduce our work.

• Download data: download_index, download query encoder, download trec_eval
• Evaluate open-sourced checkpoints: run_retrieve.sh.

Could you please follow the guidelines in the readme and see whether run_retrieve.sh will give you the expectant numbers?

Jingtao

[Rachel-guo]

Hi Jingtao,

Thanks for the improved repository and the results file. We can confirm that your improved repository can be reproduced using your code.

Buruo

[jingtaozhan]

Great!
Looking forward to see a PyTerrier plugin for JPQ!