Reproducing the result on Winogrande

[BunnyNoBugs]

Hi there!

I am trying to reproduce the results reported in WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale, namely the score of RoBERTa trained and evaluated on Winogrande. Right now I am dissatisfied with the score I get and I suspect that the thing is with how I train the model. Do I get it right that the training objective for Winogrande was multiple choice of two options based on the sentence with _? So, is this correct that the input to RoBERTa is a plain sentence with _ and two options?

Also, would you mind sharing training details such as maximum sequence length and best hyperparameters found by grid-search?

Thanks!

[keisks]

Hi, thank you for your questions!

Do I get it right that the training objective for Winogrande was multiple choice of two options based on the sentence with _? So, is this correct that the input to RoBERTa is a plain sentence with _ and two options?

You can find how the input is processed below.

winogrande/scripts/utils.py

Line 136 in 727e837

class WinograndeProcessor(DataProcessor):

Also, would you mind sharing training details such as maximum sequence length and best hyperparameters found by grid-search?

It's answered in #2