Hi! This repository contains the datasets introduced in the paper ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning.
ScoNe-NLI contains contrast sets of six examples where entailment relations are impacted by the scope of one or two negations. The six splits have 1202 examples each:
no_negation.csv: No negation is present.one_not_scoped.csv: One negation that does not scope over the relevant lexical item.two_not_scoped.csv: Two negations, neither scope over the relevant lexical item.two_scoped.csv: Two negations, one scopes over the relevant lexical item, but the second scopes over the first, canceling it out.one_scoped.csv: One negation that scopes over the relevant lexical item. This split contains the negated examples (NMoNLI) from Geiger et al., 2020.one_scoped_one_not_scoped.csv: Two negations, but only one scopes over the relevant lexical item.
In each file, premises and hypotheses are the columns labeled sentence1_edited and sentence2_edited. The entailment relations are in the gold_label_edited column. One exception is the one_scoped.csv file, where they are just labeled as sentence1, sentence2, and gold_label.
ScoNe-NLG is a natural language generation dataset that contains 74 contrasting triplets of examples where half-completed naturalistic narratives that have different coherent completions depending on the presence and scope of a negation.
English
@inproceedings{she-etal-2023-scone,
title = "{S}co{N}e: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning",
author = "She, Jingyuan S. and
Potts, Christopher and
Bowman, Samuel R. and
Geiger, Atticus",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-short.154",
pages = "1803--1821",
abstract = "A number of recent benchmarks seek to assess how well models handle natural language negation. However, these benchmarks lack the controlled example paradigms that would allow us to infer whether a model had truly learned how negation morphemes semantically scope. To fill these analytical gaps, we present the Scoped Negation NLI (ScoNe-NLI) benchmark, which contains contrast sets of six examples with up to two negations where either zero, one, or both negative morphemes affect the NLI label. We use ScoNe-NLI to assess fine-tuning and in-context learning strategies. We find that RoBERTa and DeBERTa models solve ScoNe-NLI after many shot fine-tuning. For in-context learning, we test the latest InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning. To better understand this result, we extend ScoNe with ScoNe-NLG, a sentence completion test set that embeds negation reasoning in short narratives. Here, InstructGPT is successful, which reveals the model can correctly reason about negation, but struggles to do so on NLI examples outside of its core pretraining regime.",
}