Skip to content

[RSS 2022 Best Paper, Best Student Paper Finalist] Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects

License

Notifications You must be signed in to change notification settings

real-stanford/irp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Iterative Residual Policy

This repository contains the source code for the paper Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. This paper has been accepted to RSS 2022.

drawing

Cite this work

@inproceedings{chi2022irp,
  title={Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects},
  author={Chi, Cheng and Burchfiel, Benjamin and Cousineau, Eric and Feng, Siyuan and Song, Shuran},
  booktitle={Proceedings of Robotics: Science and Systems (RSS)},
  year={2022}
}

Datasets

IRP Rope Dataset [required for eval] (7.63GB)

IRP Cloth Dataset [training only] (938MB)

Pretrained Models

IRP Rope [action + tracking] (914MB)

IRP Cloth [action] (450MB)

Usage

Installation (🖥️ Basic)

A conda environment.yml for python=3.8, pytorch=1.9.0 and cudatoolkit=11.2 is provided.

conda env create --file environment.yml

Please try mambaforge for better dependency conflict resolution.

mamba env create --file environment.yml

Installation (👾 Simulation)

Installation (🦾 Real Robot)

Hardware

Software

Evaluation (🖥️ Basic)

Under project root (i.e. irp/), create data folder and download IRP Rope Dataset as well as Pretrained Models. Extract tar files

$ cd data
$ tar -xvf checkpoints.tar
$ tar -xvf irp_rope.zarr.tar

Activate environment

$ conda activate irp
(irp) $

Run dataset evaluation script. Use action.gpu_id to select GPU on multi-GPU systems.

(irp) $ python eval_irp_rope_dataset.py action.gpu_id=0

A log.pkl file will be saved to the ad-hoc output directory created by hydra. Add command-line argument offline=False to enable wandb logging (recommended).

Numbers reported in our paper is generated using this method.

Evaluation (👾 Simulation)

Extract data and checkpoints following basic (required). Install dependencies following sim installation.

Run simulation evaluation script.

(irp) $ python eval_irp_rope_dataset.py action.gpu_id=0

Note that this method has not been extensively tested and is mainly provided for future development. Mujoco might crash due to numerical instability (i.e. Nan), which is better handled in the dataset.

Evaluation (🦾 Real Robot)

Extract data and checkpoints following basic (required). Install dependencies following real installation.

Initialize UR5 robot and write down <robot_ip>. Move robot in teach mode close to joint configuration [-90,-70,150,-170,-90,-90] to prevent unexpected movement.

Run ur5_camera_calibration_app to create homography calibration file to data/calibration.pkl. Use A/D keys to move j2 and W/S keys to move j3. Click on the tip of the wooden extension for calibration.

(irp) $ python ur5_camera_calibration_app.py --ip <robot_ip> -o data/calibration.pkl

Example for calibration result, the red and blue crosses should be fairly close. drawing

Run real eval script.

(irp) $ python eval_irp_rope_real.py

Result and videos will be saved to the hydra ad-hoc output directory.

Training

To train IRP model from scratch:

(irp) $ python train_irp.py

In case of inaccurate tracking, use video_labeler.py to generate tracking labels and train_tracker.py to train tracking model.

About

[RSS 2022 Best Paper, Best Student Paper Finalist] Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published