Fixed XNLI prompts entailment direction #52
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently there are two XNLI prompts that ask "does Sentence 2 entail Sentence 1". Note that for entailment, we can only prompt “does S1 entail S2” (does the premise entail the hypothesis) not the other way around. For contradiction, we could maybe either prompt "does S1 contradict S2" or “does S2 contradict S1”, but I sticked to the former to be consistent. Would be an interesting experiment to see if it generalizes to the latter though.
I also fixed some typos and standardized line breaks to match Figure G7 (ANLI) of the GPT-3 paper. If they look good, I will write more prompts from other papers and copy and paste them to other NLI datasets.