-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
XQUAD results reproducability for mBERT #8
Comments
Hi Max, |
Thanks, Sebastian! Interesting to see if there were differences in hparams that caused such a the difference. I can immediately see that several choices are hardcoded in
So unless you manually changed these values in the google's script these are some of the notable differences. Meanwhile, I would like to also additionally confirm that the issue is only related to |
Thanks for the note, Max. Yes, these are some of the settings that should probably explain the difference in performance. |
@maksym-del, @sebastianruder line 53 and line 63 Lines 50 to 71 in 5d7e462
Lines 59 to 66 in 5d7e462
|
Here are the results I got on XQuAD
|
@Liangtaiwan Hi, I only find test data in |
@hit-computer You can find the labels here. https://github.com/deepmind/xquad |
@Liangtaiwan Thank you very much! |
Hi @hit-computer, I've answered in the corresponding issue. Please don't post in other unrelated issues but instead tag people in your issue. |
Closing this issue. Please re-open if needed. |
Hi, thanks for the benchmark and the accompanied code!
I am trying to replicate XQUAD scores from the (XTREME) paper using this repo's code.
I run
mBert cased
model with default parameters and strictly follow the instruction in the README file.However, the results for some languages are much lower than the scores from the paper.
In particular for
vi
andth
the gap is two-fold. There is also a significant drop forhi
andel
.The e.g.
en, es, de
result, on the other hand, is comparable.Below I provide a table with scores that I just obtained from running the code together with the corresponding numbers from the paper. @sebastianruder, could I please ask you to take a look at it.
The text was updated successfully, but these errors were encountered: