Skip to content
Corpora for evaluating NLU services (like API.ai, RASA, Microsoft LUIS, ...)
Branch: master
Clone or download

Latest commit

Latest commit 9bac08b Feb 21, 2019

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
AskUbuntuCorpus.json training-flag added to AskUbuntu Corpus Oct 22, 2017
ChatbotCorpus.json training-flag added to Chatbot Corpus Aug 29, 2018
LICENSE License added Jul 3, 2017
README.md added errate to readme Feb 21, 2019
WebApplicationsCorpus.json lang tag added Oct 18, 2017

README.md

README

This project is a collection of three corpora which can be used for evaluating chatbots or other conversational interfaces. Two of the corpora were extracted from StackExchange, one from a Telegram chatbot.

If you use the data and publish please let us know and cite our SIGdial 2017 paper:

@InProceedings{braun-EtAl:2017:SIGDIAL,
  author    = {Braun, Daniel  and  Hernandez-Mendez, Adrian  and  Matthes, Florian  and  Langen, Manfred},
  title     = {Evaluating Natural Language Understanding Services for Conversational Question Answering Systems},
  booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue},
  month     = {August},
  year      = {2017},
  address   = {Saarbrücken, Germany},
  publisher = {Association for Computational Linguistics},
  pages     = {174--185},
  url       = {http://www.aclweb.org/anthology/W17-3622}
}

Errata

There is an error in Table 5 of the paper. In the "true +" column, the overall sum should be 573, not 820, and accordingly precision, recall, and f-score are 0.92, 0.85, and 0.88.

[The reason for this error is in the Excel evaluation sheet, the total number of "true +" (573) was stored as number of "true +" for the chatbot corpus. Added up with the result for the other corpora (77, 170) we end up with 820.]

License

All three corpora are released under the CC BY-SA 3.0 license.

Content

Ask Ubuntu Corpus

162 questions and answers from https://askubuntu.com.

Five intents (MakeUpdate, SetupPrinter, ShutdownComputer, SoftwareRecommendation, None) and three entity types (Printer, Software, Version).

Web Applications Corpus

89 questions and answers from https://webapps.stackexchange.com.

Eight intents (ChangePassword, DeleteAccount, DownloadVideo, ExportData, FilterSpam, FindAlternative, SyncAccounts, None) and three entity types (WebService, OS, Browser).

Chatbot Corpus

206 questions from a Telegram chatbot for public transport in Munich.

Two intents (Departure Time, Find Connection) and five entity types (StationStart, StationDest, Criterion, Vehicle, Line).

Evaluation Scripts

Python scripts for automated evaluation are provided here.

Contact Information

If you have any questions, please contact:

Daniel Braun (Technical University of Munich) daniel.braun@tum.de

You can’t perform that action at this time.