Natural Language Instruction GENeration
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

NLIGEN - Natural Language Instruction GENeration

This repository contains the codebase for Daniele et al. HRI 2017 paper.


The folder datasets/ contains both the original and the augmented version of the SAIL dataset (collected by MacMahon et al., 2006).

CAS Language

The CAS language used in these datasets is slightly different from the one proposed by MacMahon et al., 2006. In particular, some attributes of some CAS actions (e.g., Turn, Travel) were removed (or updated) since they were not used in the original dataset. The file CAS-specifications.pdf shows the structure of the new CAS language. Both datasets are written using the new CAS language.

Ho to use the datasets

Both datasets are organized in a hierarchy of examples (demonstrations), each of which is associated with a unique ID called ex_id.

Original SAIL Dataset

Info about the original SAIL dataset:

    Python Pickle (v.2)
    ~1 MB
Number of demonstrations: 
    698 (3213 total segments)

    A demonstration in the original dataset contains:
    - the name of the world (map) in which it is defined;
    - a path modeled as a sequence of poses;
    - a list of cas-instruction pairs, one for each segment of the path;
    Each segment represents a section of the path corresponding to an entire natural language instruction paragraph. For each example, a progressive number called 'seg_id' identifies distinct segments of the path.

    D = Dataset
    D[<ex_id>]['map'] = name of the world (i.e., Grid, L, Jelly) for the demonstration <ex_id>.
    D[<ex_id>]['segments'][<seg_id>]['english'] = english paragraph describing the segment <seg_id>.
    D[<ex_id>]['segments'][<seg_id>]['cas'] = list of CAS command strings describing the segment <seg_id>.
    D[<ex_id>]['segments'][<seg_id>]['path'] = sequence of poses - (x,y,theta) tuples - for <seg_id>.

Augmented SAIL Dataset

Info about the augmented SAIL dataset:

    Python Pickle
    480 MB
Number of demonstrations:

    A demonstration in the augmented dataset is simply a cas-instruction pair;
    D = Dataset
    D[<ex_id>]['segments'][<seg_id>] = set of cas-instruction pairs generated from (<ex_id>,<seg_id>).
    D[<ex_id>]['segments'][<seg_id>][k]['english'] = k-th artificially generated english paragraph.
    D[<ex_id>]['segments'][<seg_id>][k]['cas'] = k-th list of artificially generated CAS command strings.

The augmented dataset was generated by using only 560 demonstrations from the original dataset. The remaining 138 demonstrations are used for testing and do not need to be augmented. Compare the ex_ids in both datasets to recover the set of non-augmented demonstrations.



  • OS: Linux / MacOSX / Windows
  • Python 2.7

The package code/CAS contains a class for each action, sub-task, and description object defined by the CAS language (see file CAS-specifications.pdf).

The code in code/CAS/ defines all the possible directions, distances, objects, floor patterns, wall pictures that can be used to populate the fields of the CAS commands.

The package code/utils contains utility functions.

The script code/ contains examples of how to load and parse the demonstrations in the datasets.


If you use any of our data, please cite our paper using the following BibTex entry:

  title={Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation},
  author={Daniele, Andrea F and Bansal, Mohit and Walter, Matthew R},
  booktitle={Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)},