LeTO: Learning Constrained Visuomotor Policy with Differentiable Trajectory Optimization

Zhengtong Xu, Yu She

Purdue University

arXiv | Summary Video

---

LeTO uniquely integrates a differentiable trajectory optimization layer for visuomotor policy learning, enabling generating actions in a safe and controlled fashion.

---

Codebase

The code is adapted from diffusion policy and robomimic.

For details of LeTO and the differentiable trajectory optimization, please see LeTO/model/LeTO.py.

---

Installation

We use the same installation process as diffusion policy. Please refer to this instruction for a detailed installation guide and use this yaml file. In addition, please also install diffusion policy in letoenv.

For downloading the simulation training datasets, please refer to this instruction.

Start Training

Activate conda environment and login to wandb (if you haven't already).

$ conda activate letoenv
$ wandb login

For example, launch training with seed 42 on GPU 0.

$ python train_LeTO.py --config-dir=./LeTO/config --config-name=train_LeTO_robomimic_square_mh.yaml training.seed=42 training.device=cuda:0 hydra.run.dir='data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}'

Checkpoints

Checkpoints and training logs of LeTO can be accessed via this Google drive link.