No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Attending to objects for robot learning.

This code provides an interface to the attention mechanism described in "Deep Object-Centric Representations forGeneralizable Robot Learning" (Devin et al. 2017) available on arxiv It uses a tensorflow model to attend to objects and it publishes the results to ROS at about 10-20Hz on our machines. This is intended to be used with an robot learning package such as Guided Policy Search (

This code is written for Python2 and Tensorflow 1.2, and ROS Indigo. For dependencies, we recommend using a python virtualenv to avoid conflicts with other pip installs. This is done in the install script:


Other platforms are not yet supported.

To use the example data, copy it into the data directory:

mkdir taskdata
mkdir taskdata/pouring
cd taskdata/pouring
tar -xvf pouringdata.tar.gz
rm pouringdata.tar.gz
cd ../..

Now we will select a crop of the object to initialize the features.

python example.yaml
python  taskdata/pouring/myimage.png myfeats.npy

The script will display an image form the demo with the RPN boxes. Click on a pixel to select the box that contains it. If multiple box contains the pixel you clicked, use the left/right arrow keys to cycle through them. Try click on the brown mug and press ENTER when your preferred box is green. This will save out the features to myfeats.npy.

If you don't want to do any finetuning, you can just use "myfeats.npy" as the attention. However, to finetune follow the following steps:

python example.yaml
python example.yaml -i myfeats.npy

The model will save out weights periodically, to reload the network from iteration 10000 and look at it's attention, run

python example.yaml -t 10000

This will open up an IPython notebook and you can view the soft-attended box by running

To instead save the attention, run

python example.yaml -s 10000

which will save it in the experiment directory.

Finally, to publish the attention to ros, run:

python taskdata/pouring/myexperiment/attention_queries.npy

Acknowledgements: We thank Ronghang Hu for porting RPN from to tensorflow. This work was done with the support of Huawei Technologies and the National Science Foundation.