Multi-Modality Driven Impedance-Based Sim2Real Transfer Learning for Robotic Multiple Peg-in-Hole Assembly

• Quick Demos

video_assembly.mp4

Installation

• Run conda create -n dynamic-assembly python=3.8.13 && conda activate dynamic-assembly to create and activate a new python environment.
• Install MuJoCo using these instructions (i.e. extract the downloaded mujoco210 directory into ~/.mujoco/mujoco210)
• Use obj2mjcf to process original robot urdf file and assembly-related object file into XML file for the use in MuJoCo.
• Run cd dynamic-assembly && ./install-dependencies.sh to install all required dependencies.

Training a Pretrained Vision model

• Collect a dataset from different hole/peg shapes during raw policy learning based on the ShuffleNet-v2 perception. An example dataset of circle shape can be obtained https://drive.google.com/drive/folders/181-17Ub87fH-swqxKHRSpU4iK7TywIp7?usp=drive_link.
• The pretarining framework is designed from simple VAE models. You can test the collected dataset based on these models.

Construction of Simulation Environment

• Modified from the robosuite, we use our own Kuka_iiwa robot and self-designed peg/hole objects to execute the assembly task. Model files are in robosuite/models and the assembly environment settings in robosuite/enviroments.
• The manipulation of robot model and robot controllers are based on the MujocoPy.

Visualization of training process

• Install the package of Viskit from rllab.

Our Network module from SAC algorithm

• Install our modified rlkit package from rlkit.

Updating...