DEDO - Dynamic Environments with Deformable Objects

DEDO is a lightweight and customizable suite of environments with deformable objects. It is aimed for researchers in the machine learning, reinforcement learning, robotics and computer vision communities. The suite provides a set of every day tasks that involve deformables, such as hanging cloth, dressing a person, and buttoning buttons. We provide an example for integrating StableBaselines3 for reinforcement learning, as well as reference implementations for training various Variational Autoencoder variants with our environment. DEDO is easy to set up and has few dependencies, it is highly parallelizable and supports a wide range of customizations: loading custom objects and textures, adjusting material properties.

For a brief overview, please see our intro video. For more details please see the paper.

@inproceedings{dedo2021,
  title={Dynamic Environments with Deformable Objects},
  author={Rika Antonova and Peiyang Shi and Hang Yin and Zehang Weng and Danica Kragic},
  booktitle={Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
  year={2021},
}

Table of Contents:
Installation
GettingStarted
Tasks
Use with RL
Use with VAE
Customization

Please refer to Wiki for the full documentation

Installation

The project is managed with uv and targets Python 3.10+.

git clone https://github.com/contactrika/dedo
cd dedo
uv sync --all-extras   # creates .venv, installs locked deps + dev tools

macOS build note

On recent macOS versions, pybullet 3.2.7 fails to compile due to a conflict between its bundled zlib and the system SDK headers. Work around this by setting CFLAGS before the initial sync:

CFLAGS="-Wno-error=implicit-function-declaration -Dfdopen=fdopen" uv sync --all-extras

The project already configures no-build-isolation-package = ["pybullet"] in pyproject.toml so that pybullet is built with numpy available (required for getCameraImage to return numpy arrays).

Video recording

To enable recording/logging videos install ffmpeg:

# Linux
sudo apt-get install ffmpeg
# macOS
brew install ffmpeg

See more in Installation Guide in wiki

Getting started

To get started, one can run one of the following commands to visualize the tasks through a hard-coded policy.

uv run python -m dedo.demo --env=HangGarment-v1 --viz --debug

• dedo.demo is the demo module
• --env=HangGarment-v1 specifies the environment
• --viz enables the GUI
• --debug outputs additional information in the console
• --cam_resolution 400 specifies the size of the output window

See more in Usage-guide

Tasks

See more in Task Overview

We provide a set of 10 tasks involving deformable objects, most tasks contains 5 handmade deformable objects. There are also two procedurally generated tasks, ButtonProc and HangProcCloth, in which the deformable objects are procedurally generated. Furthermore, to improve generalzation, the v0 of each task will randomizes textures and meshes.

All tasks have -v1 and -v2 with a particular choice of meshes and textures that is not randomized. Most tasks have versions up to -v5 with additional mesh and texture variations.

Tasks with procedurally generated cloth (ButtonProc and HangProcCloth) generate random cloth objects for all versions (but randomize textures only in v0).

HangBag

uv run python -m dedo.demo_preset --env=HangBag-v1 --viz

HangBag-v0: selects one of 108 bag meshes; randomized textures

HangBag-v[1-3]: three bag versions with textures shown below:

HangGarment

uv run python -m dedo.demo_preset --env=HangGarment-v1 --viz

HangGarment-v0: hang garment with randomized textures (a few examples below):

HangGarment-v[1-5]: 5 apron meshes and texture combos shown below:

HangGarment-v[6-10]: 5 shirt meshes and texture combos shown below:

HangProcCloth

uv run python -m dedo.demo_preset --env=HangProcCloth-v1 --viz

HangProcCloth-v0: random textures, procedurally generated cloth with 1 and 2 holes.

HangProcCloth-v[1-2]: same, but with either 1 or 2 holes

Buttoning

uv run python -m dedo.demo_preset --env=Button-v1 --viz

ButtonProc-v0: randomized textures and procedurally generated cloth with 2 holes, randomized hole/button positions.

ButtonProc-v[1-2]: procedurally generated cloth, 1 or two holes.

Button-v0: randomized textures, but fixed cloth and button positions.

Button-v1: fixed cloth and button positions with one texture (see image below):

Hoop

uv run python -m dedo.demo_preset --env=Hoop-v1 --viz

Hoop-v0: randomized textures Hoop-v1: pre-selected textures

Lasso

uv run python -m dedo.demo_preset --env=Lasso-v1 --viz

Lasso-v0: randomized textures Lasso-v1: pre-selected textures

DressBag

uv run python -m dedo.demo_preset --env=DressBag-v1 --viz

DressBag-v0, DressBag-v[1-5]: demo for -v1 shown below

Visualizations of the 5 backpack mesh and texture variants for DressBag-v[1-5]:

DressGarment

uv run python -m dedo.demo_preset --env=DressGarment-v1 --viz

DressGarment-v0, DressGarment-v[1-5]: demo for -v1 shown below

Mask

uv run python -m dedo.demo_preset --env=Mask-v1 --viz

Mask-v0, Mask-v[1-5]: a few texture variants shown below:

HangGarmentRobot

uv run python -m dedo.demo_preset --env=HangGarmentRobot-v1 --viz

HangGarmentRobot-v1: A environment for demonstrating integration with Franka Robot Arm

FoodPacking

uv run python -m dedo.demo_preset --env=FoodPacking-v1 --viz

FoodPacking-v[0-3]: Demonstrating robotic manipulation of pushing YCB objects

Rendering Point Cloud Observations

DEDO now supports pointcloud observations. Use the flag --pcd to set the environment observations to return point clouds. They are segmented based on the active rigid and deformable object.

Example usage, visual demo

uv run python -m dedo.demo --env=HangGarment-v1 --viz --debug --pcd --logdir rendered
uv run python -m dedo.demo --env=HangBag-v1 --viz --debug --pcd --logdir rendered

Example usage, preset trajectory demo

uv run python -m dedo.demo_preset --env=HangGarment-v1 --viz --debug --pcd --logdir rendered
uv run python -m dedo.demo_preset --env=HangBag-v1 --viz --debug --pcd --logdir rendered

Known issues:

• PyBullet can only segment the deformable object if it has ID=0. We assume this to be true, and load the deformable object first. However, this seems to cause the floor to disappear in the visual render.

RL Examples

dedo/run_rl_sb3.py gives an example of how to train an RL algorithm with Stable-Baselines3:

uv run python -m dedo.run_rl_sb3 --env=HangGarment-v0 \
    --logdir=/tmp/dedo --num_play_runs=3 --viz --debug

Note: RLlib integration (run_rllib.py) is currently broken. It depends on the pre-2.0 ray.rllib.agents API which has been removed in modern Ray. The [rllib] extra is still declared but non-functional.

For documentation, please refer to Arguments Reference page in wiki

To launch Tensorboard:

tensorboard --logdir=/tmp/dedo --bind_all --port 6006 \
  --samples_per_plugin images=1000

SVAE Examples

dedo/run_svae.py gives an example of how to train various flavors of VAE:

uv run python -m dedo.run_svae --env=HangGarment-v0 \
    --logdir=/tmp/dedo --viz --debug

To launch Tensorboard:

tensorboard --logdir=/tmp/dedo --bind_all --port 6006 \
  --samples_per_plugin images=1000

Customization

To load custom object you would first have to fill an entry in DEFORM_INFO in task_info.py. The key should the the .obj file path relative to data/:

DEFORM_INFO = {
...
    # An example of info for a custom item.
    'bags/custom.obj': {
        'deform_init_pos': [0, 0.47, 0.47],
        'deform_init_ori': [np.pi/2, 0, 0],
        'deform_scale': 0.1,
        'deform_elastic_stiffness': 1.0,
        'deform_bending_stiffness': 1.0,
        'deform_true_loop_vertices': [
            [0, 1, 2, 3]  # placeholder, since we don't know the true loops
        ]
    },

Then you can use --override_deform_obj flag:

uv run python -m dedo.demo --env=HangBag-v0 --cam_resolution 200 --viz --debug \
    --override_deform_obj bags/custom.obj

For items not in DEFORM_DICT you will need to specify sensible defaults, for example:

uv run python -m dedo.demo --env=HangGarment-v0 --viz --debug \
  --override_deform_obj=generated_cloth/generated_cloth.obj \
  --deform_init_pos 0.02 0.41 0.63 --deform_init_ori 0 0 1.5708

Example of scaling up the custom mesh objects:

uv run python -m dedo.demo --env=HangGarment-v0 --viz --debug \
   --override_deform_obj=generated_cloth/generated_cloth.obj \
   --deform_init_pos 0.02 0.41 0.55 --deform_init_ori 0 0 1.5708 \
   --deform_scale 2.0 --anchor_init_pos -0.10 0.40 0.70 \
   --other_anchor_init_pos 0.10 0.40 0.70

See more in Customization Wiki

Additonal Assets

BGarment dataset is adapter from Berkeley Garment Library

Sewing dataset is adapted from Generating Datasets of 3D Garments with Sewing Patterns