New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can not reproduce results in the paper for WN18RR dataset #15
Comments
There has been a major bug in the code-base found by Victoria Lin. I fixed the issue here: d830ddf I have not reproduced all of the results, but in general, they are the same or similar. What I have gotten so far on WN18RR is: Current code: MRR: 0.43, HIts@10: 0.51, Hits@3: 0.44, Hits@1: 0.39 The difference is now: I would say that these scores are even slightly better than we got before, but of course, the MRR is lower. I think I would be able to replicate the results if I would search for longer (the score increased steadily), but I currently have no time to do this. I updated the results in the README.md with the current scores. If you get better scores please let me know. I got these scores with:
|
I have run a WN18RR network with the same parameters for a bit longer, I got these results:
I will run a wider grid search and see if I can improve the MRR. |
Could you explain what kind of bug? |
Please see #18 for a detailed explanation of the bug and ongoing re-evaluation of ConvE. |
I have tried the generic command for reproducing results in the paper for WN18RR dataset, but it could not reproduce MRR reported in the paper, I managed only to get
0.42
.Which hyperparameters can reproduce the
0.46
MRR reported in the paper?The text was updated successfully, but these errors were encountered: