Benchmarking of Natural Language Understanding models in context of service robots

This repository contains the code, datasets, and suplemental materials for "Tell Your Robot What To Do: Evaluation of Natural Language Models for Robot Command Processing" by Erick Romero Kramer, Argentina Ortega Sainz, Alex Mitrevski, and Paul G. Ploeger.

Models used for Benchmarking

• Mbot - Repository
• Rasa - Documentation - Repository
• LU4R - Project Site
• ECG - Repository

Repository structure

• code/
  • ecg_framework_code
    • Required repository for ECG
  • ecg_grammars
    • Contains grammars generated for our Benchmarking
  • ecg_robot_code
    • Modified version of ECG to extract the semantic specifications
  • ecg_workbench_release
    • Tool used to generated ECG grammar files
  • lu4r
    • Empty folder
  • rasa_models
    • Contains the nlu models and scripts to generate the training datasets
  • mbot
    • Contains the trained classifiers and scripts to generate the training datasets
• datasets/
  • GPSR category 1 (Single and multiple actions)
  • GPSR category 2 (Single and multiple actions)
  • ROPOD
• docs
  • RnD Report
    • Detailed information about the related work presented in the paper.
    • Detailed guideline about the models selected (how the work, how to train/adapt them, and how to use them).
    • Explicit description of the process followed during the benchmarking.
  • paper.pdf - PENDING

Notes:

• Pending to add link to the published version of the paper.
• Due to space limitations, the code can be downloaded from here

Demo:

• Navigation Commands using Mbot - Video

Author

Erick Romero Kramer - email - University email