Natural Language Understanding (NLU) using Deep Neural Networks (DNN)

Maps robot commands (text) into intentions and arguments used during RoboCup@Home 2018

Map robot commands given as text (speech recognition is not covered) into "intention and arguments".

Example:

• Go to the kitchen, intention: go, args: kitchen is a destination
• Please drive to the kitchen, intention: go, args: kitchen is a destination
• Robot navigate to the kitchen now, intention: go, args: kitchen is a destination

This allows robots to be flexible at understanding human commands given with small variations.

Brought to you by:

• Instituto Superior Tecnico, Lisboa
• Institute for Systems and Robotics (ISR)
• Laboratory of Robotics and Engineering Systems (LARSyS)
• SocRob RoboCup team

This code was tested on Ubuntu 16.04 and ROS kinetic. The DNN was trained using TensorFlow platform.

The code is organized in 4 packages:

• mbot_nlu (inference) : Given a pretrained model (which can be generated with mbot_nlu_training), this package contains ROS wrapper and ROS independent python code to use the DNN and map command text into intention + args.
• mbot_nlu_classifiers : A place to store pretrained models (generated by mbot_nlu_training)
• mbot_nlu_filter : Tool to remove unwanted characters such as #'?$%, etc. as well as to identify specific questions that the robot should directly respond without the need of a DNN.
• mbot_nlu_training : Contains scripts to generate training samples + a tensor flow implementation to train a LSTM Neural Network. DNN training is completely ROS independent and is based on python 3.

Minimum hardware requirements (Inference / Training)

• Processor - Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz / Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
• Memory - 2GB RAM / 5GB RAM
• Graphics - Not required / NVIDIA GEFORCE GTX 1060
• OS - Tested under Ubuntu 16.04 and ROS Kinetic

DNN Inference setup

• Clone this repository to your workspace source folder using the below command

git clone https://github.com/socrob/mbot_natural_language_processing.git  
cd mbot_natural_language_processing

• Install dependencies and finish the setup by invoking the automated script (require admin privilages)

sudo ./repository.debs
(Give No for all the unnecessary prompts)

• Build the packages and source the workspace

catkin build mbot_nlu mbot_nlu_classifiers mbot_nlu_filter mbot_nlu_training
source ../../devel/setup.bash

DNN Inference

Single sentence classification

• Run the launch file

roslaunch mbot_nlu mbot_nlu.launch nlu_classifier:=mithun_gpsr_robocup

• Visualize the intention and arguments

rostopic echo /hri/nlu/mbot_nlu/output_recognition

• Trigger the node with some text

rostopic pub --once /hri/nlu/mbot_nlu/input_sentence std_msgs/String "data: 'go to the kitchen'"

Multi sentence classification

• Set ros parameter to use sentence divider

roscd mbot_nlu
gedit ros/launch/mbot_nlu.launch
set "use_syntaxnet" param to "True"
save and close the file

• Run the launch file

roslaunch mbot_nlu mbot_nlu.launch nlu_classifier:=mithun_gpsr_robocup

• Visualize the intention and arguments

rostopic echo /hri/nlu/mbot_nlu/output_recognition

• Trigger the node with some text

rostopic pub --once /hri/nlu/mbot_nlu/input_sentence std_msgs/String "data: 'go to the kitchen and pick the spoon'"

DNN training setup

• Clone this repository to your PC dedicated for training the DNN using the below command

git clone https://github.com/socrob/mbot_natural_language_processing.git  
cd mbot_natural_language_processing

• Install dependencies and finish the training setup by invoking the automated script

source mbot_nlu_training/common/setup/nlu_training_setup.sh

• Edit and Verify the parameters used in the training using the configuration file

gedit mbot_nlu_training/ros/config/config_mbot_nlu_training.yaml

• Generate the data and initiate the training
  • Intention classifier training

  cd mbot_nlu_training/common/src/mbot_nlu_training/gpsr/intention_NN/
  ./gpsr_data_generator.py (wait for completion)
  ./train_nn_model.py
  

  • Arguments/ Slots classifier training

  cd mbot_nlu_training/common/src/mbot_nlu_training/gpsr/slots_NN/
  ./gpsr_data_generator.py (wait for completion)
  ./train_nn_model.py
  

Upon training completion the classifiers will saved in latest_intents_classifier and latest_slots_classifier in the respective folders (intention_NN/slots_NN). Copy the files in the above mentioned folders to mbot_nlu_classifiers/common/classifiers/ with sample structure given below and follow the DNN Inference procedure with nlu_classifier parameter as your CLASSIFIER_NAME

Your classifier should be in a tree such as:

root
     | - CLASSIFIER_NAME
           | - intent
           | - slots

NLU automated test

To execute the test, simply execute the python script in the test folder. The test input sentences are sourced from nlu_test_inputs.txt and corresponding ideal outputs are sourced from nlu_expected_output.txt files. Both files can be modified to the user needs, but make sure that the input sentence and the corresponding output are properly paired. Once the test is complete a detailed log of errors will be published in log_file.txt in the same folder.

cd mbot_natural_language_processing/mbot_nlu/common/test
./mbot_nlu_test.py

RoboCup Command Generator bulk data processor

A script has been added to process the bulk generated data from the RoboCup command generator into clean, sorted, unique and simple list of phrases. A sample source file is also added in the same directory.

cd mbot_natural_language_processing/mbot_nlu_training/common/scripts
./process_raw.py