No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.



RocketScience is a prototype for an innovative platform that allows laymen to be guided doing tasks in a hands-free manner. It uses the Azure Luis service to hold conversations with a user and detect the users intents in natural spoken language. It includes a user feedback loop to rate the instructions steps on the fly.

Packages to install

To run the "Rocket Science" script you have to install

  • PyAudio and SpeechRecognition (to detect spoken language in Python locally)
  • simpleaudio (to play sound file by Bing)
pip3 install -r requirements.txt

Microsoft Azure Services

This Service is hooked up to Microsoft Azure. You need the following services:

  • Bing Speech (to do text to speech conversion)
  • Azure Luis (to detect intends in the user's commands) A sample of utterances has to be provided at (url applies if you are operating a service for the EU)

You will need api keys for that. To get those go to

These keys shall then be stored in a json file in the folder auth/ which should look like this:


The Luis utterance database is stored in luis_model.json.


A database with all the instructions has to be stored in constants.DB_NAME. There is a small default sqlite table provided in duesentrieb.db You can also create your own mock table by executing:


The content of this small sample Database can be seen in the files content.csv and structure.csv


To start the listening server, execute:


The user can then via a speech interface send search requests to the instructions database for hands-free assistance on reqested tasks. Once a task is selected the user can talk RocketScience through the instruction steps. After each step the user can rate the instructions given and ask for a different version (if one is provided)