Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Neural Conversational QA

This repository contains the script and data used for the EMNLP 2020 paper titled Neural Conversational QA: Learning to Reason vs Exploiting Patterns [ArXiV] [ACL].

Repository structure

The repository is arranged as follows. The datasets folder contains all datasets used in the paper. The file names are of the form <dataset>_<suffix>.json where dataset may be one of history, mod or sharc representing the History-Shuffled dataset (Section 4), the ShARC-Mod dataset (Section 5) or the official ShARC datasets as mentioned in the paper. suffix may be one of train, dev and dev_multi representing the training set, the dev set, and the dev set with multiple references (Section 3, Appendix A.3).

├── datasets                            # all datasets used in the paper
├── requirements.txt
├──                              # wrapper script to recreate datasets
├── scripts                             # scripts to create modified datasets
│   ├──
│   ├──
│   ├──
│   ├──
│   └──                    # the heuristic based program (Section 2.1)
└── sharc1-official                     # the official ShARC dataset

The scripts folder has scripts to recreate the datasets, as well as the heuristic based program (Section 2.1). sharc1-official is directly downloaded from, and requirements.txt has all dependencies necessary to run the scripts.

Recreating datasets

if you wish to use the scripts to recreate the datasets used in the paper, the script should be useful. Please make sure you use the correct version of the dependencies. Not doing so might lead to different resutls. We recommend that you use conda as described below.

$ conda create -n py36 python=3.6
$ conda activate py36
$ pip install -r requirements.txt
$ python3 -m spacy download en_core_web_sm
$ python3 -m spacy download en_core_web_md
$ bash


If you find this repository useful, kindly use the following.

    title = "Neural Conversational {QA}: Learning to Reason vs Exploiting Patterns",
    author = "Verma, Nikhil  and
      Sharma, Abhishek  and
      Madan, Dhiraj  and
      Contractor, Danish  and
      Kumar, Harshit  and
      Joshi, Sachindra",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "7263--7269",
    abstract = "Neural Conversational QA tasks such as ShARC require systems to answer questions based on the contents of a given passage. On studying recent state-of-the-art models on the ShARC QA task, we found indications that the model(s) learn spurious clues/patterns in the data-set. Further, a heuristic-based program, built to exploit these patterns, had comparative performance to that of the neural models. In this paper we share our findings about the four types of patterns in the ShARC corpus and how the neural models exploit them. Motivated by the above findings, we create and share a modified data-set that has fewer spurious patterns than the original data-set, consequently allowing models to learn better.",


Our heuristic model scripts/ contains code derived from the preprocessing script in The evaluation script scripts/ is derived from the official evaluation script


Repository for the EMNLP 2020 paper on Neural Conversational QA



No releases published


No packages published