Fixed XNLI prompts entailment direction

[awebson]

Currently there are two XNLI prompts that ask "does Sentence 2 entail Sentence 1". Note that for entailment, we can only prompt “does S1 entail S2” (does the premise entail the hypothesis) not the other way around. For contradiction, we could maybe either prompt "does S1 contradict S2" or “does S2 contradict S1”, but I sticked to the former to be consistent. Would be an interesting experiment to see if it generalizes to the latter though.

I also fixed some typos and standardized line breaks to match Figure G7 (ANLI) of the GPT-3 paper. If they look good, I will write more prompts from other papers and copy and paste them to other NLI datasets.

[VictorSanh]

oh yes, I messed up (I always talk about entailment but never use the verb entail though...) thank you @awebson !

I think these choices on line breaks, including "question" or not, including a comma or not are arbitrary and we are looking for this diversity. Ultimately, a model shouldn't be sensitive to how the prompt+x is formulated
But I hear your concern on matching exactly the format in gpt3

I'll merge your pr

[awebson]

Thanks! Yeah sometimes I also have to pause for a second and think, wait, which one is the premise and which one is the hypothesis again? I forgot which paper started doing it but they renamed P and H to just “context” and “question”, which are much less confusing.

I 100% agree with you that we’d like the model to be robust to line breaks and stuff. GPT-3 is still weirdly sensitive to them (e.g., bottom left paragraph at p. 6 of Reynolds & McDonell 2021), and you’re right that having diverse punctuations in the training set may improve invariance.