ACL2020 Tutorial: Open-Domain Question Answering

This ACL2020 tutorial will be was held on July 5th, 2020, by Danqi Chen <danqic@cs.princeton.edu> and Scott Yih <scottyih@fb.com>. You can find all the tutorial materials below and the video is available now.

Overview

This tutorial provides a comprehensive and coherent overview of cutting-edge research in open-domain question answering (QA), the task of answering questions using a large collection of documents of diversified topics. We will start by first giving a brief historical background, discussing the basic setup and core technical challenges of the research problem, and then describe modern datasets with the common evaluation metrics and benchmarks. The focus will then shift to cutting-edge models proposed for open-domain QA, including two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever-free methods. Finally, we will cover some hybrid approaches using both text and large knowledge bases and conclude the tutorial with important open questions. We hope that the tutorial will not only help the audience to acquire up-to-date knowledge but also provide new perspectives to stimulate the advances of open-domain QA research in the next phase.

A more detailed introduction can be found here.

Tutorial Slides

1. Introduction & problem definition
2. A history of open-domain (textual) QA
3. Datasets & evaluation
4. Two-stage retriever-reader approaches
5. Dense retriever and end-to-end training
6. Retrieval-free approaches
7. Open-domain QA using KBs and text
8. Conclusion

Reading List

Early QA work

1. Answering English questions by computer: a survey. R.F.Simmons. 1965
2. The Structure and Performance of an Open-domain Question Answering System. Dan Moldovan, Sanda Harabagiu, Marius Pasca, Rada Mihalcea, Roxana Girju, Richard Goodrum, Vasile Rus. ACL 2000
3. An Analysis of the AskMSR Question-answering System. Eric Brill, Susan Dumais and Michele Banko. EMNLP 2002.
4. Open-Domain Question–Answering. John Prage. 2007
5. An Exploration of the Principles Underlying Redundancy-Based Factoid Question Answering. Jimmy Lin. 2007.
6. Building Watson: An Overview of the DeepQA Project. David Ferrucci, Eric Brown, Jennifer Chu-Carroll et al. 2010

Recent work (2017+)

1. Reading Wikipedia to Answer Open-Domain Questions. Danqi Chen, Adam Fisch, Jason Weston, Antoine Bordes. ACL 2017.
2. R^3: Reinforced Reader-Ranker for Open-Domain Question Answering. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, Jing Jiang. AAAI 2018.
3. Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, Murray Campbell. ICLR 2018.
4. Denoising Distantly Supervised Open-domain Question Answering. Yankai Lin, Haozhe Ji, Zhiyuan Liu, Maosong Sun. ACL 2018.
5. Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William Cohen. EMNLP 2018.
6. Language Models are Unsupervised Multitask Learners. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. OpenAI 2019.
7. End-to-end Open-domain Question Answering with BERTserini. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, Jimmy Lin. NAACL 2019 (demonstration).
8. Latent Retrieval for Weakly Supervised Open Domain Question Answering. Kenton Lee, Ming-Wei Chang, Kristina Toutanova. ACL 2019.
9. Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P. Parikh, Ali Farhadi, Hannaneh Hajishirzi. ACL 2019.
10. Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader. Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, William Yang Wang. ACL 2019.
11. A Discrete Hard EM Approach for Weakly Supervised Question Answering. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, Luke Zettlemoyer. EMNLP 2019.
12. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, Bing Xiang. EMNLP 2019.
13. PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text. Haitian Sun, Tania Bedrax-Weiss, William W. Cohen. EMNLP 2019.
14. Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering. Sewon Min, Danqi Chen, Luke Zettlemoyer, Hannaneh Hajishirzi. arXiv 2019.
15. Contextualized Sparse Representations for Real-Time Open-Domain Question Answering. Jinhyuk Lee, Minjoon Seo, Hannaneh Hajishirzi, Jaewoo Kang. ACL 2020.
16. Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong. ICLR 2020.
17. REALM: Retrieval-Augmented Language Model Pre-Training. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. ICML 2020.
18. Dense Passage Retrieval for Open-Domain Question Answering. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. EMNLP 2020.
19. How Much Knowledge Can You Pack Into the Parameters of a Language Model?. Adam Roberts, Colin Raffel, Noam Shazeer. EMNLP 2020.
20. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. NeurIPS 2020.
21. Language Models are Few-Shot Learners. Tom B. Brown, Benjamin Mann, Nick Ryder et al. NeurIPS 2020.

We understand this is a long reading list :) In case you wonder where you should start with, we plan to discuss the papers in bold in depth during our tutorial.

Latest work (2020+, partially updated)

1. AmbigQA: Answering Ambiguous Open-domain Questions. Sewon Min, Julian Michael, Hannaneh Hajishirzi, Luke Zettlemoyer. EMNLP 2020.
2. The 3rd Workshop on Machine Reading for Question Answering. EMNLP 2021.
3. Evaluation Paradigms in Question Answering. Pedro Rodriguez and Jordan Boyd-Graber. EMNLP 2021.
4. Joint Passage Ranking for Diverse Multi-Answer Retrieval. Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, Hannaneh Hajishirzi. EMNLP 2021.
5. SituatedQA: Incorporating Extra-Linguistic Contexts into QA. Michael J.Q. Zhang and Eunsol Choi. EMNLP 2021 (Outstanding Paper).
6. Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation. EMNLP 2021.
7. You Only Need One Model for Open-domain Question Answering. Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christoper D. Manning, Kyoung-Gu Woo. Preprint 2021.
8. Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks. Akari Asai*, Matt Gardner*, Hannaneh Hajishirzi. Preprint 2021.

Miscellaneous

1. Billion-scale similarity search with GPUs (a.k.a. faiss).
2. Lilian Weng - How to Build an Open-Domain Question Answering System?