K-Scale Manipulation Suite

Gymnasium+MuJoCo Environments

KManipSoloArm environment has one 7dof arm with a 1dof gripper. KManipSoloArmVision has a gripper cam, a head cam, and an overhead cam.	KManipDualArm environment has two 7dof arms with 1dof grippers. KManipDualArmVision has 2 gripper cams, a head cam, and an overhead cam.	KManipTorso environment has a 2dof head, two 6dof arms with 1dof grippers. KManipTorsoVision has 2 gripper cams, a head cam, and an overhead cam.

Setup

clone and install dependencies

git clone https://github.com/kscalelabs/gym-kmanip.git && cd gym-kmanip
conda create -y -n gym-kmanip python=3.10 && conda activate gym-kmanip
pip install -e .

run tests to verify installation

pip install pytest
pytest

for installation on the edge, just install on bare metal

sudo apt-get install libhdf5-dev
git clone https://github.com/kscalelabs/gym-kmanip.git && cd gym-kmanip
pip install -e .

Getting Started

visualize one of the mujoco environments using the viewer

python gym_kmanip/examples/1_view_env.py

alternatively mujoco provides a nice standalone visualizer

download standalone mujoco

tar -xzf ~/Downloads/mujoco-3.1.5-linux-x86_64.tar.gz -C /path/to/mujoco-3.1.5
/path/to/mujoco-3.1.5/bin/simulate gym_kmanip/assets/_env_solo_arm.xml

Recording Data

data can be recorded to a rerun .rrd file for visualization

python gym_kmanip/examples/record_to_rrd.py
rerun gym_kmanip/data/foo/episode_1.rrd

data can be recorded to a h5py .hdf5 file for training models

python gym_kmanip/examples/record_to_hdf5.py

you can also record data to a video file for sharing on social media

python gym_kmanip/examples/record_to_mp4.py

Teleop

teleop can be used to control the robot and optionally record datasets

🤗 K-Scale HuggingFace Datasets

you will need additional dependencies

pip install vuer==0.0.30

start the server on the robot computer

python gym_kmanip/examples/4_record_data_teleop.py

start ngrok on the robot computer.

ngrok http 8012

open the browser app on the vr headset and go to the ngrok url

Real Robot

the real robot works just like any other environment

you will need additional dependencies

pip install opencv-python==4.9.0.80

Help Wanted

✅ solo arm w/ vision

✅ dual arm w/ vision

✅ torso w/ vision

✅ inverse kinematics using mujoco

⬜️ tune and improve ik

⬜️ recording dataset via teleop

⬜️ training policy from dataset

⬜️ evaluating policy on robot

Dependencies

• Gymnasium is used for environment
• MuJoCo is used for physics simulation
• PyTorch is used for model training
• Rerun is used for visualization
• H5Py is used for logging datasets
• HuggingFace is used for dataset & model storage
• Vuer teleop only is used for visualization
• ngrok teleop only is used for networking

helpful links and repos

• dm_control
• gym-pybullet-drones
• gym-aloha
• lerobot
• universal_manipulation_interface
• loco-mujoco

Citation

@misc{teleop-2024,
  title={gym-kmanip},
  author={Hugo Ponte},
  year={2024},
  url={https://github.com/kscalelabs/gym-kmanip}
}