Skip to content

vanyacohen/beo-language-grounding

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

Grounding Language Attributes to Objects using Bayesian Eigenobjects

Code and the dataset for reproducing the experiments of the paper “Grounding Language Attributes to Objects using Bayesian Eigenobjects” (IROS 2019). https://arxiv.org/abs/1905.13153

Setup

The repository needs several large dataset files, which can be downloaded here: https://drive.google.com/drive/folders/1_6AdIbaEpOdvTo2kg4GG9z8ApKRVCs1i

data/

Datasets are organized into car, couch, and plane respectively. Data files are equivalent across the object classes.

car/

Contains BEO and language annotation data for each object in the cars class.

Language data files:

Format: object_id, attr_1, attr_ 2, attr_3, attr_4, attr_5, attr_6, natural_language_description

car_train.csv

car_dev.csv

car_test.csv

Contains 10 annotations per object_id, with attribute ratings from 1-5, and a natural language description of the object. Objects are sourced from the shapenet.org project.

BEO vector files:

Format: object_id, numpy_float_array

car_full_obv_vecs_300.csv Contains the fully-observed BEO vectors for each object in the dataset.

limited_viewpoint_car_partial_view_train.csv Contains BEO vectors obtained from partially observed front-facing views of objects using the techniques in “Hybrid Bayesian Eigenobjects: Combining Linear Subspace and Deep Network Methods for 3D Robot Vision” (HBEO)

limited_viewpoint_car_partial_view_test.csv Contains a disjoint test-set of BEO vectors obtained for objects from side-rear-facing views of objects using HBEO.

partial_view_car_vectors_300.csv Contains BEO vectors obtained from partial observations of objects, from all angles using HBEO.

partial_view_test_car_vectors_300.csv Contains a disjoint test-set of BEO vectors for objects obtained from partial observations of objects, from all angles using HBEO.

docs/

Contains language annotation for each class' attributes and attribute ratings.

training/

nlmodel.py Contains the language grounding models (Bag-of-Words & Embedding Model). EmbedModel was the language model used in the paper.

nlretnn.py Contains the training and evaluation script for the full-view experiment (with fully-observed BEO vectors).

usage: nlretnn.py [-h] [--beo_size BEO_SIZE] [--traindata TRAINDATA]
                  [--testdata TESTDATA] [--devdata DEVDATA]
                  [--objvectors OBJVECTORS] [--model_output MODEL_OUTPUT]

optional arguments:
  -h, --help             show this help message and exit
  --beo_size BEO_SIZE    size of each image dimension
  --traindata TRAINDATA  data file
  --testdata TESTDATA    data file
  --devdata DEVDATA      data file
  --objvectors OBJVECTORS
                         beo vectors
  --model_output MODEL_OUTPUT
                         model output name

nlretnn_partial.py Contains the training and evaluation script for the partial-view and view-transfer experiments (with partially-observed BEO vectors).

usage: nlretnn_partial.py [-h] [--beo_size BEO_SIZE] [--traindata TRAINDATA]
                          [--testdata TESTDATA] [--devdata DEVDATA]
                          [--testvectors TESTVECTORS]
                          [--trainvectors TRAINVECTORS]
                          [--model_output MODEL_OUTPUT]

optional arguments:
  -h, --help             show this help message and exit
  --beo_size BEO_SIZE    size of each image dimension
  --traindata TRAINDATA  data file
  --testdata TESTDATA    data file
  --devdata DEVDATA      data file
  --testvectors TESTVECTORS
                         beo vectors
  --trainvectors TRAINVECTORS
                         beo vectors
  --model_output MODEL_OUTPUT
                         model output name

Citing

If you use our dataset or code please cite:

@inproceedings{cohen2019grounding,
  title={Grounding Language Attributes to Objects using Bayesian Eigenobjects},
  author={Cohen*, Vanya and Burchfiel*, Benjamin and Nguyen*, Thao and Gopalan, Nakul and Tellex, Stefanie and Konidaris, George},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2019},
  month={November}
}

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages