About reproducing

[selimfirat]

Thank you for sharing this wonderful work!
In fact, I have succeeded to reproduce with higher scores :) This situation makes me wonder why this happens. Was there any project/data update or am I doing something wrong?

I followed the "To run with local or virtual environment" instructions in README.py without changing config.yaml since it says "to recreate the experiment from paper, leave as is". I obtain an absolute 3% more recall.

I obtain 0.87 precision and 0.83 recall, whereas in the paper, it is 87.5% precision and 80.0% recall.

[khundman]

@selimfirat thanks for the note, I was able to reproduce this as well, and the paper is incorrect for this particular experiment. I plan to issue a revised version of the paper with this correction and another that has been found, but not sure on timing of when I'll get to that.