Skip to content
Python scripts for evaluating NLU services (like API.ai, RASA, Microsoft LUIS, ...)
Python
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
nlu_analysers
nlu_converters
LICENSE
README.md
example.py

README.md

NLU-Evaluation-Scripts

Python scripts for automatically evaluating NLU services (API.ai, IBM Watson Conversation, Microsoft LUIS, RASA) based on the format used by NLU-Evaluation-Corpora.

Training

You can use the converters to create .json- or .zip-files with training data which can be imported into the respective NLU service using the web interface.

Microsoft Luis

#luis (also works for rasa)
luis_converter = LuisConverter()
luis_converter.import_corpus("WebApplicationsCorpus.json")
luis_converter.export("WebApplicationsTraining_Luis.json")

You can also use the Luis file format to train Rasa, however, we recommend using the Dialogflow data format for training Rasa.

IBM Watson Conversation

#watson
watson_converter = WatsonConverter()
watson_converter.import_corpus("WebApplicationsCorpus.json")
watson_converter.export("WebApplicationsTraining_Watson.json")

Dialogflow (formerly known as API.ai)

#dialogflow (also works for rasa)
dialogflow_converter = DialogflowConverter()
dialogflow_converter.import_corpus("WebApplicationsCorpus.json")
dialogflow_converter.export("WebApplicationsTraining_Dialogflow.zip")

Evaluation

You can use the analysers to annotate the test data and generate a .json-file with an evaluation of the annotations.

Microsoft Luis

#luis
luis_analyser = LuisAnalyser("application_id", "subscription_key")
luis_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Luis.json")
luis_analyser.analyse_annotations("WebApplicationsAnnotations_Luis.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Luis.json")

IBM Watson Conversation

#watson
watson_analyser = WatsonAnalyser("workspace_id", "user", "password")
watson_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Watson.json")
watson_analyser.analyse_annotations("WebApplicationsAnnotations_Watson.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Watson.json")

Dialogflow (formerly known as API.ai)

#dialogflow
dialogflow_analyser = DialogflowAnalyser("project_id")
dialogflow_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Dialogflow.json")
dialogflow_analyser.analyse_annotations("WebApplicationsAnnotations_Dialogflow.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Dialogflow.json")

Rasa NLU

In order to evaluate the annotations from Rasa, you have to start the Rasa server with the option "-e luis".

#rasa
rasa_analyser = RasaAnalyser("http://localhost:5000/parse")
rasa_analyser.get_annotations("WebApplicationsCorpus.json", "WebApplicationsAnnotations_Rasa.json")
rasa_analyser.analyse_annotations("WebApplicationsAnnotations_Rasa.json", "WebApplicationsCorpus.json", "WebApplicationsAnalysis_Rasa.json")

Contact Information

If you have any questions, please contact:

Daniel Braun (Technical University of Munich) daniel.braun@tum.de

You can’t perform that action at this time.