New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
why I can't reach your performance of baseline? #6
Comments
What results did you get? I would suggest deleting the cache files and reruning everything from scratch (make sure you follow the instructions closely). I ran through the process once and found the results reproducible. |
hi @kimiyoung. I also ran the entire codebase following the instructions. This was from a clean clone and building of dataset. I got only a best dev F1 of 56.462817452313814. I ran this a couple of times after that and it seems like the score is about 56+. EM is about 42.3+. Any ideas on what might be the cause? Thanks! |
I got slightly better results, best_dev_F1 56.881756072546665. I too did it from scratch |
I believe it is a matter of variance. AFAIK, there could be three factors that led to this:
I would suggest trying different random seeds to study the effects of model variance. Some random seeds might work better. |
@kimiyoung Thanks for your reply! Actually I tried both versions (with and without 100). Im guessing maybe it's an issue with system or dependencies. I'll try different seeds. I have one question though, in your early experiments did you try different optimizers or just defaulted to SGD right from the start? Thanks! |
@vanzytay I did not try other optimizers. |
I got a even worse result... |
1080Ti best_dev_F1 57.83286201117724 |
@kimiyoung Thanks for your work. Sure, will try to use other random seeds. P.S. following are the results from the default run - Evaluation: {'sp_em': 0.1950033760972316, 'joint_recall': 0.3910371172630571, 'f1': 0.5661927280885037, 'recall': 0.5830912961848933, 'joint_f1': 0.36950188461400907, 'sp_f1': 0.6090896536879039, 'joint_prec': 0.4142776503762301, 'em': 0.42822417285617825, 'sp_recall': 0.624765441625671, 'prec': 0.589656389075701, 'sp_prec': 0.664514002765185, 'joint_em': 0.09790681971640783} |
After the update(V1.1), i got a acceptable result. |
I got best_dev_F1 56.37454881285825 (on 2080ti) |
why I can't reach your performance of baseline?
The text was updated successfully, but these errors were encountered: