Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Visual Foresight

Code for reproducing experiments in Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control, along with expanded support for additional robots.

On a high level, Visual Model Predictive Control (visual-MPC) leverages an action-conditioned video prediction model (trained from unsupervised interaction) to enable robots to perform various tasks with only raw-pixel input. This codebase provides an implmentation of: unsupervised data collection, our benchmarking framework, the various planning costs, and - of course - the visual-MPC controller! Additionally, we provide: instructions to reproduce our experiments, Dockerfiles for our simulator environments, and documentation on our Sawyer robot setup.

Crucially, this codebase does NOT implement video prediction model training, or meta-classifier model training. If you're only interested in training models, please refer to Stochastic Adversarial Video Prediction and/or Few-Shot Goal Inference for Visuomotor Learning and Planning.


General Dependencies

Since this project is deployed in sim and on a robot, all code is written to be compatible with Python 2.7 and Python 3.5.


Manual Installation

Our simulator requires Python 3.5.2 and MuJoCo 1.5 to run successfully. We strongly recommend using a virtual environment (such as Anaconda) for this project. After you setup Python and MuJoCo, installation directions are as follows:

# install video prediction code
git clone && cd video_prediction-1 && git checkout dev && python develop && cd ..
# install meta-classifier code
git clone
#install visual-MPC
git clone && cd visual_foresight
pip install -r requirements.txt
python develop

Docker Installation

Docker allows a cleaner way to get started with our code. Since we heavily use the GPU, you will have to install nvidia-docker and all related dependencies. After that run:

git clone && cd docker && cp ~/.mujoco/mjkey.txt ./
nvidia-docker build -t foresight/sim:latest .

Now to create a new bash in this environment run: nvidia-docker run -it foresight/sim: bash


Hardware Setup

All experiments are conducted on a Sawyer robot with an attached WSG-50 gripper. The robot is filmed from two orthogonal viewing angles using consumer webcams. Refer to the paper for further details.

Software Setup

Robot code heavily uses ROS. Assuming you use our same hardware, you will need to install the following:

Once you've installed the dependencies:

  1. Clone our repository into your ROS workspace's src folder. Then run catkin_make to rebuild your workspace.
  2. Clone and install the video_prediction code-base.
  3. Clone and install the meta-classifier code-base
  4. Remember to install our python packages by running sudo python develop in EVERY project workspace
  5. Start up required ROS nodes:
# in a new intera terminal
roslaunch foresight_rospkg start_cameras.launch   # run cameras

# in a new intera terminal
roslaunch foresight_rospkg start_gripper.launch   # start gripper node

# in a new intera terminal
roscd foresight_rospkg/launch
rosrun foresight_rospkg     # (optional) stop after Sawyer recognizes the gripper

Experiment Reproduction

In sim, data collection and benchmarks are started by running python visual_mpc/sim/ The correct configuration file must be supplied, for each experiment/data collection run. Similarly, rosrun foresight_rospkg is the primary entry point for the robot experiments/data-collection.

Data Collection

By default data is saved in the same directory as the corresponding python config file. Rollouts are saved as a series of pickled dictionaries and JPEG images, or as compressed TFRecords.


Use run_robot to start random data collection on the Sawyer.

  • For hard object collection: rosrun foresight_rospkg <robot name/id> data_collection/sawyer/hard_object_data/ -r
  • For deformable object collection: rosrun foresight_rospkg <robot name/id> data_collection/sawyer/towel_data/ -r


Use visual_mpc/sim/ to start random data collection in our custom MuJoCo cartgripper environment

  • To collect data with l-block objects and autograsp (x, y, z, wrist rotation, grasp reflex) action space run: python visual_mpc/sim/ data_collection/sim/grasp_reflex_lblocks/ --nworkers <num_threads>

Convert to TFRecords

While the raw (pkl/jpeg file) data format is convenient to work with, it is far less efficient for model training. Thus, we offer a utility in visual_mpc/utils/ which converts data from our raw format to compressed TFRecords.

Running Benchmarks

Again pass in the python config file to the corresponding entry point. This time add a --benchmark flag!


  • For Registration Experiments: rosrun foresight_rospkg <robot name/id> experiments/sawyer/registration_experiments/ --benchmark
  • For Mixed Object Experiments (one model which handles both deformable and rigid objects)
    • Rigid: rosrun foresight_rospkg <robot name/id> experiments/sawyer/mixed_objects/ --benchmark
    • Deformable: rosrun foresight_rospkg <robot name/id> experiments/sawyer/mixed_objects/ --benchmark
  • Meta-classifier experiments are coming soon


Coming soon!

Pretrained Models

Coming soon


If you find this useful, consider citing:

  title={Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control},
  author={Ebert, Frederik and Finn, Chelsea and Dasari, Sudeep and Xie, Annie and Lee, Alex and Levine, Sergey},
  journal={arXiv preprint arXiv:1812.00568},


Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control




No packages published