Skip to content


Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network

Venue: ICRA 2019

Author's mail :,

This package produces visually similar robot hand poses based on depth images of the human hand in an end-to-end fashion, which is a collaborative work done by TAMS and Fucun Sun's lab of Tsinghua University.

The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and from simulated depth images of a robotic hand.

Please cite this paper (Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network), if you use our released code.



Installation Instructions


  • [ROS Kinetic and Ubuntu 16.04]]
  • [CUDA 9]

ROS Dependency

Python Dependency

  • python3.8 (for the and the other pytorch trainning)
  • python2.7 (for the ROS stuff and,, and
  • PyTorch
  • numpy
  • tensorboard
  • matplotlib
  • pickle
  • pandas
  • seaborn
  • numba
  • rospkg
  • opencv-python

Camera Drive

  • librealsense


  • Install necessary packages (kinetic branch) for Shadow Hand.
  • Install Bio IK packages. Please follow the Basic Usage in in bio_ik repository and set correct kinematics solver.
  • Install RealSense Camera package:
    sudo apt install ros-kinetic-realsense-camera
  • To simplify it, you can put above packages in one ros workspace.
  • Download our package in same workspace, then build this package with catkin_make.

Dataset Generation

  • Download (BigHand2.2M dataset). Put the lable file Training_Annotation.txt into ros/src/shadow_teleop/data/Human_label/. Building nine folders respectively called depth_shadow[1-8] saves robot depth images from nine viewpoint inros/src/shadow_teleop/data/.
  • Generate robot mapping file by human hand keypoints from BigHand2.2M dataset. The generated file save in ros/src/shadow_teleop/data/human_robot_mapdata.csv.
    python ros/src/shadow_teleop/scripts/
  • Run shadow hand in gazebo and use the our simulation world (./ros/src/teleop_motorhand/worlds/
    roslaunch teleop_motorhand gazebo.launch
  • Generate dataset by running the code:
    roslaunch shadow_teleop multi_shadow_sim_bio.launch
    Please note the location of saved depth images and the location of robot_joints_file.csv.
  • Save robot_joints_file.csv as joint_all.npy by pandas.readcsv() or numpy.loadtxt().
  • Crop human hand images into 100*100 (normalized to [0,255)):
      python ros/src/shadow_teleop/scripts/
      python utils/
  • Crop shadow images into 100*100(normalized to [0,255)):
      python utils/
    Please change the location of original depth images and cropped depth images fl to your own dataset location.
  • Last but not least, spilt joint_all.npy into training dataset and test dataset by yourself, and save as joint_train.npy and joint_test.npy at your own dataset location.

Model Training

  • If you want to train the network yourself instead of using a pretrained model, follow below steps.

  • Launch a tensorboard for monitoring:

    tensorboard --log-dir ./assets/log --port 8080

    and run an experiment for 200 epoch:

    python --epoch 200 --mode 'train' --batch-size 256 --lr 0.01 --gpu 1 --tag 'teachnet' --data-path 'LOCATION OF YOUR TRAINING DATASET'

    File name and corresponding experiment:                    --- Teach Hard-Early approach     --- Single human    --- Single shadow                --- Teach Soft-Early approach

Pretrained Models:

RealsenseF200 Realtime Demo

  • Launch camera RealsenseF200 (If you use the other camera which is suitable for close-range tracking, please use corresponding launch file). Or you can download the recorded example rosbag, and play the bag file:

    roslaunch realsense2_camera rs_rgbd.launch
    rosbag play [-l] example.bag
  • Limit your right hand to the viewpoint range of [30°, 120°] and the distance range of [15mm, 40mm] from the camera.

  • Change the correct topic name in based on your camera.

  • Run the testing of TeachNet on python3 enviroment

    python [--model-path pretrained-model-location --cuda --gpu 0]

Demo in simulation

  • Run Shadow hand in simulation
    roslaunch teleop_motorhand demo.launch
  • Run the demo code on python2 enviroment

Demo in real world.

We provide safe mode demo and unsafe mode demo ( and The unsafe mode uses the SrHandCommander and doesn't check collision, so the response of the robot is fast and low latency.

  • Run the real robot

Safe mode:

  • Run the collision check service:
    rosrun shadow_teleop interpolate_traj_service
  • Run the demo code on python2 enviroment

Unsafe mode:

  • Run the demo code on python2 enviroment


If you use this work(collobrated with ), please cite:

  title={Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network},
  author={Li, Shuang and Ma, Xiaojian and Liang, Hongzhuo and G{\"o}rner, Michael and Ruppel, Philipp and Fang, Bing and Sun, Fuchun and Zhang, Jianwei},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},


Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network






No releases published


No packages published