-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
XQuAD can have a better evaluation #28
Comments
Scores before changing the script
Scores after changing the script
|
Thanks for the note. I had experimented with using the MLQA evaluation script for XQuAD but only observed marginal differences in some experiments (as mentioned here). If the differences are indeed larger, we might consider updating the evaluation script. What model did you use to obtain the scores? |
Hi @sebastianruder, I used However, as I mentioned in #8 (comment), there is a bug in scripts/*_qa.sh. |
I'm sure that the difference is large on Chinese. The result is reported in https://arxiv.org/pdf/1909.09587.pdf. |
When evaluating XQuAD, it uses the original SQuAD evaluation scripts.
However, the script does not consider the tokenization for Chinese. (MLQA does)
ex. "天氣很好"(The weather is good.) should be tokenized to "天", "氣", "很", "好" or "天氣", "很好"
I'm sure that it will get a more convincing score on Chinese, but I'm not sure it's better on others or not.
I modify the original code to the following
Maybe you can consider it?
The text was updated successfully, but these errors were encountered: