Pretrained language models are becoming commonplace in a wide variety of applications. Although these systems are highly accurate there is room for significant improvement in tasks such as text entailment (TE) in real-world situations. For example, we want to make sure these models perform well when dealing with non-native English speakers who might use unusual constructions or place their sentences out-of-order. In this research, we have measured the overall performance of RoBERTa ([1]) when sentence pairs are switched (sentence 1 -> sentence 2 and sentence 2 -> sentence 1) while retaining the same label and have further analyzed the model’s response where the originally assigned label was contradiction. We find that RoBERTa doesn’t perform well in certain types of sentence pairs (that are contradictory) due to linguistic complexities (ambiguity for example) in English language. We are particularly focused on those switched labels that predict entailment from contradiction. We consider such switching in case of mission critical applications such as health care, emergency, security, privacy and terrorism has very dire consequences.
Our research highlights the importance of building language models that take into account linguistic complexities and provide future recommendations on model improvements with respective contradictory sentence pairs. We also conclude the importance of creating better benchmark datasets that take into account these linguistic ambiguities. Finally we challenge the notion that English sentence pairs can only be categorized with 3 different labels - Neutral, Entailment and Contradiction.