Add XNLI train set

[lhoestq]

I added the train set that was built using the translated MNLI.
Now you can load the dataset specifying one language:

from datasets import load_dataset

xnli_en = load_dataset("xnli", "en")
print(xnli_en["train"][0])
# {'hypothesis': 'Product and geography are what make cream skimming work .', 'label': 1, 'premise': 'Conceptually cream skimming has two basic dimensions - product and geography .'}
print(xnli_en["test"][0])                                                                                                                   
# {'hypothesis': 'I havent spoken to him again.', 'label': 2, 'premise': "Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again."}

Cc @sgugger

[YifanYangEbidanko]

Hi! Thanks for adding the translated MNLI! Do you know what translations system / model you used when you created the datasets in the other languages?

[lhoestq]

According to the paper it's the result of the work of professional translators ;)

[YifanYangEbidanko]

Thanks for getting back to me. The training data is not from translators. And it appears to be machine translation for all languages. If we can know what system was used to create the training data that would be great! Yifan.

…

On Thu, Jun 9, 2022, 05:51 Quentin Lhoest ***@***.***> wrote: According to the paper <https://arxiv.org/pdf/1809.05053.pdf> it's the result of the work of professional translators ;) — Reply to this email directly, view it on GitHub <#781 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKLKWDAPTMGB6BE5GJ4GULVOG5BLANCNFSM4TE67NMQ> . You are receiving this because you commented.Message ID: ***@***.***>

[lhoestq]

The training data is not from translators.

What makes you think that ? The paper litteraly says

we hire translators to translate the resulting sentences into 15 languages using the One Hour Translation platform.

[YifanYangEbidanko]

However the annotators only did test and validation sets, as this was what in the paper: “we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages".

…

On Thu, Jun 9, 2022 at 10:35 AM Quentin Lhoest ***@***.***> wrote: The training data is not from translators. What makes you think that ? The paper litteraly says we hire translators to translate the resulting sentences into 15 languages using the One Hour Translation platform. — Reply to this email directly, view it on GitHub <#781 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKLKWFZOQPLK4WSKFRLW6DVOH6LLANCNFSM4TE67NMQ> . You are receiving this because you commented.Message ID: ***@***.***>