Skip to content
Train a robot to see the environment and autonomously perform different tasks
Branch: master
Clone or download
Latest commit 6c783e1 Nov 22, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
autoencoder Update Oct 15, 2017
demonstrate_ros_package Add files via upload Oct 11, 2017
.gitignore main commit Jul 12, 2017
LICENSE Initial commit Jul 12, 2017 Update Jul 14, 2017 Update Nov 22, 2017 some cleanup Jul 14, 2017 some cleanup Jul 14, 2017 main commit Jul 12, 2017 some cleanup Jul 14, 2017 main commit Jul 12, 2017


This framework is designed to demonstrate arbitrary object manipulation tasks to a robot and train a deep neural network to autonomously mimic that behavior by processing sequence of images from the environment.


Videos of task demonstration and autonomous performance by a Lynxmotion AL5D robot


  • Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, Sergey Levine. Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration. arXiv





First you need a robot (If you do not have a robot, you can try the simulator in this repository: roboinstruct-1). The system should work for different robots. However, the code in this repository is only able to control a Lynxmotin AL5D robot arm. Connect this robot to your computer. Connect a camera and set its corresponding image topic in and Place the camera in a position so that it can see the robot arm and the relevant objects in the environment. Install either PlayStation Move API or LeapMotion driver and connect the device to your computer. Copy demonstrate_ros_package folder to your ROS workspace and build the ros package (catkin_make etc.). For PS Move, change the path to the PS Move API in file. Run these commands:

rosrun demonstrate_leap leap_al5d -l 1 (1 for LeapMotion, 0 for PS Move)

rosrun demonstrate_leap (Only for PS Move)

rosrun demonstrate_leap

If everything goes well, the recorded demonstration will be saved in your home directory. Copy them into a new folder in roboinstruct root folder and call it trajectories. Create another folder in roboinstruct root folder and call it models.

Learning reads the data from trajectories folder and converts it into appropriate formats for Chainer or Blocks code to train the neural network. Open file, change make_dataset_mode to prepare_data_for_chainer, and run Go to autoencoder/chainer_vaegan/src and run When the vision part is trained well after a few epochs, copy the saved models from autoencoder/chainer_vaegan/model to autoencoder/chainer_vaegan/model-saved folder. Now in file, change make_dataset_mode to prepare_data_for_blocks, and run run to train the LSTM model given visual features. This saves the final model into the models folder.

Most of the parameters can be set in There are some ideas implemented that are not included in the paper such as multi-timescale learning, auxiliary predictions, etc. You can take a look at the to see a short description of these ideas. If you feel confident, you can change these parameters.

Testing on the robot

For testing the trained model on the robot, first run rosrun demonstrate_leap leap_al5d so that you can send commands to the robot. Setup the camera and run it to publish the images on its corresponding ROS topic. Set the ID of the task that you want to robot to perform in task_to_perform variable in Run to load the saved model and generates the trajectory and send commands to the robot.


The codes in the char-rnn, chainer-vae-gan, and leap-hand repositories helped a lot in this project.

You can’t perform that action at this time.