Multi-turn response selection in retrieval-based chatbots is a task which aims to select the best-matched response from a set of candidates, given the context of a conversation. This task is attracting more and more attention in academia and industry. However, no one has maintained a leaderboard and a collection of popular papers and datasets yet. The main objective of this repository is to provide the reader with a quick overview of benchmark datasets and the state-of-the-art studies on this task, which serves as a stepping stone for further research.
- Lowe et al. (2015) released the original version. [Download]
- Xu et al. (2016) released a version in which numbers, paths and URLs were replaced by placeholders. [Download]
- Gu et al. (2019) released a version based on Xu et al. (2016) in which contexts and responses are assigned ids to track them. [Download]
- Lowe et al. (2017) released the original version. [Download]
- Gu et al. (2019) released a version in which contexts and responses are assigned ids to track them. [Download]
- Wu et al. (2017) released the original version. [Download]
- Gu et al. (2019) released a version in which contexts and responses are assigned ids to track them. [Download]
- Zhang et al. (2018) released the original version. [Download]
- Gu et al. (2019) released a version in which contexts and responses are assigned ids to track them. [Download]
| Model | R_2@1 | R_10@1 | R_10@2 | R_10@5 | Paper and Code |
|---|---|---|---|---|---|
| IMN (Gu et al., 2019) | 0.945 | 0.771 | 0.886 | 0.979 | Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots |
| U2U-IMN (Gu et al., 2019) | 0.943 | 0.762 | 0.877 | 0.975 | Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots |
| HRDE-LTC (Yoon et al., 2018) | 0.915 | 0.652 | 0.815 | 0.966 | Learning to Rank Question-Answer Pairs using Hierarchical Recurrent Encoder with Latent Topic Clustering |
| Model | R_10@1 | R_10@2 | R_10@5 | Paper and Code |
|---|---|---|---|---|
| IMN (Gu et al., 2019) | 0.621 | 0.797 | 0.964 | Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots |
| U2U-IMN (Gu et al., 2019) | 0.616 | 0.806 | 0.966 | Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots |
| MSN (Yuan et al, 2019) | 0.606 | 0.770 | 0.937 | Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots |
| IOI (Tao et al., 2019) | 0.563 | 0.768 | 0.950 | One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues |
| DUA (Zhang et al., 2018) | 0.501 | 0.700 | 0.921 | Modeling Multi-Turn Conversation with Deep Utterance Aggregation |
| SMN (Wu et al., 2017) | 0.453 | 0.654 | 0.886 | Sequential Matching Network: A New Architecture for Multi-Turn Response Selection in Retrieval-Based Chatbots |
In addition to the studies mentioned above, there are stil a lot of great studies on multi-turn response selection worth reading. We list a part of them below.
- Dually Interactive Matching Network for Personalized Response Selection in Retrieval-Based Chatbots. Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, Quan Liu. EMNLP 2019.
- Sampling Matters! An Empirical Study of Negative Sampling Strategies for Learning of Matching Models in Retrieval-based Dialogue Systems. Jia Li, Chongyang Tao, wei wu, Yansong Feng, Dongyan Zhao, Rui Yan. EMNLP 2019.
- Multi-Turn Response Selection in Retrieval-Based Chatbots with Iterated Attentive Convolution Matching Network. Heyuan Wang, Ziyi Wu, Junyu Chen. CIKM 2019.
- Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems. Jiazhan Feng, Chongyang Tao, Wei Wu, Yansong Feng, Dongyan Zhao, Rui Yan. ACL 2019.
- Training Neural Response Selection for Task-Oriented Dialogue Systems. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su. ACL 2019.
- DSTC7 Task 1: Noetic End-to-End Response Selection. Chulaka Gunasekara, Jonathan K. Kummerfeld, Lazaros Polymenakos, Walter Lasecki. ACL 2019 Workshop.
- A Document-grounded Matching Network for Response Selection in Retrieval-based Chatbots. Xueliang Zhao, Chongyang Tao, Wei Wu, Can Xu, Dongyan Zhao, Rui Yan. IJCAI 2019.
- Sequential Attention-based Network for Noetic End-to-End Response Selection. Qian Chen, Wen Wang. AAAI 2019 Workshop on DSTC 7.
- Building Sequential Inference Models for End-to-End Response Selection. Jia-Chen Gu, Zhen-Hua Ling, Yuping Ruan, Quan Liu. AAAI 2019 Workshop on DSTC 7.
- Training Millions of Personalized Dialogue Agents. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, Antoine Bordes. EMNLP 2018.
- Personalizing Dialogue Agents: I have a dog, do you have pets too?. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston. ACL 2018.
- Learning Matching Models with Weak Supervision for Response Selection in Retrieval-based Chatbots. Yu Wu, Wei Wu, Zhoujun Li, Ming Zhou. ACL 2018.
- Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, Haiqing Chen. SIGIR 2018.
- Improving Response Selection in Multi-turn Dialogue Systems by Incorporating Domain Knowledge. Debanjan Chaudhuri, Agustinus Kristiadi, Jens Lehmann, Asja Fischer. CONLL 2018.
- Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus. Jianxiong Dong, Jim Huang. ArXiv.
- Multi-view Response Selection for Human-Computer Conversation. Xiangyang Zhou, Daxiang Dong, Hua Wu, Shiqi Zhao, Dianhai Yu, Hao Tian, Xuan Liu, Rui Yan. EMNLP 2016.
- Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. Rui Yan, Yiping Song, Hua Wu. SIGIR 2016.
Although we work very hard to list more work, the studies we select to present in this repository are by no means complete. To this end, we welcome more people to participate in the maintenance of this project. Please feel free to open issues, pull requests or contact us (gujc@mail.ustc.edu.cn).