The source code of ACL 2018 paper "Denoising Distantly Supervised Open-Domain Question Answering".
Clone or download
Latest commit e8faddc Nov 7, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
src Delete Aug 15, 2018
LICENSE Initial commit May 8, 2018 Update Nov 7, 2018 Update Nov 5, 2018


The source codes for paper "Denoising Distantly Supervised Open-Domain Question Answering", which is modified based on the code of paper " Reading Wikipedia to Answer Open-Domain Questions."


pytorch 0.3.0 numpy scikit-learn termcolor regex tqdm prettytable scipy nltk pexpect 4.2.1

Evaluation Results

Dataset Quasar-T SearchQA TrivialQA SQuAD
Models EM F1 EM F1 EM F1 EM F1
GA (Dhingra et al., 2017) 26.4 26.4 - - - -
BiDAF (Seo et al., 2017) 25.9 28.5 28.6 34.6 - - - -
AQA (Buck et al., 2017) - - 40.5 47.4 - - - -
R^3 (Wang et al., 2018a) 35.3 41.7 49 55.3 47.3 53.7 29.1 37.5
Our Model 42.2 49.3 58.8 64.5 48.7 56.3 28.7 36.6


We provide Quasar-T, SearchQA and TrivialQA dataset we used for the task in data/ directory. We preprocess the original data to make it satisfy the input format of our codes, and can be download at here.

To run our code, the dataset should be put in the folder data/ using the following format:


  • train.txt, dev.txt, test.txt: format for each line: {"question": quetion, "answers":[answer1, answer2, ...]}.

  • train.json, dev.json, test.json: format [{"question": question, "document":document1},{"question": question, "document":document2}, ...].


  • glove.840B.300d.txt: word vectors obtained from here.


  • all jar files from Stanford Corenlp.


The source codes of our models are put in the folders src/.

Train and Test

For training and test, you need to:

  1. Pre-train the paragraph reader: python --batch-size 256 --model-name quasart_reader --num-epochs 10 --dataset quasart --mode reader

  2. Pre-train the paragraph selector: python --batch-size 64 --model-name quasart_selector --num-epochs 10 --dataset quasart --mode selector --pretrained models/quasart_reader.mdl

  3. Train the whole model: python --batch-size 32 --model-name quasart_all --num-epochs 10 --dataset quasart --mode all --pretrained models/quasart_selector.mdl


If you use the code, please cite the following paper:

  1. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising Distantly Supervised Open-Domain Question Answering. In Proceedings of ACL. pages 1736--1745. [pdf]


  1. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of ACL. pages 1832--1846.

  2. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of ICLR.

  3. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830.

  4. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang,Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of AAAI. pages 5981--5988.