Author: Ritika Ghosh
ReadMEmontage.mp4
The CopyCat package enables the 4 fingered Wonick robotics’ Allegro hand to mirror the finger movement of an actual hand in ROS 2. This package has 2 modes, one where the robot hand is capable of mimicking gross finger movement while the other mode allows for the recognition of 5 types of hand gestures which are useful for carrying out finer grasping tasks. In order to perform these tasks the package utilizes and RGB camera to observe the human hand movements tracked using mediapipe’s machine learning framework to either employ an algorithm to calculate and retarget the joint states directly or use a pre-trained hand gesture recognition package to obtain already defined corresponding grasping hand configuration. These 16 joint state angles are fed to the Moveit! planner to plan and execute valid hand movements.
- Wonik Allegro Robotic Hand Version 4.0
- PCAN-USB Adapter Note: Allegro code is written for PCAN devices
The following packages are meant to be used with ROS 2 Humble.
- Make sure ROS packages are most recent and up-to-date
sudo apt update
sudo apt upgrade
- Install Moveit!:
sudo apt install ros-humble-moveit - Create a new ROS2 workspace and enter it
mkdir -p copycatws/src
cd copycatws
- Install dependencies:
sudo apt-get install cmake gcc g++ libpopt-dev
pip install media pipe
sudo apt-get install python3-opencv
- Download, build, and install PCAN-USB driver for Linux: libpcan
tar -xzvf peak-linux-driver-x.x.tar.gz cd peak-linux-driver-x.x make NET=NO sudo make install sudo modprobe pcan - Download, build, and install PCAN-Basic API for Linux: libpcanbasic
tar -xzvf PCAN_Basic_Linux-x.x.x.tar.gz cd PCAN_Basic_Linux-x.x.x/pcanbasic make sudo make install - Download, build, and install Grasping Library for Linux, "libBHand": Grasping_Library_for_Linux
- Download this package into the src directory:
git clone git@github.com:GhoshRitika/Allegro_hand.git - Download my gesture recognition fork in the same src directory:
git clone git@github.com:GhoshRitika/go1-gesture-command.git
Refer to README of individual packages for more detailed instructions and information.
- Allegro_hand
- allegro_driver : ROS2 C++
- Allegro_moveit_config
- allegro_lib: C++
- finger_tracking: ROS 2 python
- go1-gesture-command
- Connect PCAN-USB and Allegro Hand (make sure to power off Allegro Hand)
- Power on Allegro Hand using the toggle switch on the side.
- To use the CopyCat package in the first mode, i.e to visually operate the hand movement:
ros2 launch allegro_driver launch_all.launch.xml teleop:=true - To use the gesture recognition mode:
ros2 launch allegro_driver launch_all.launch.xml teleop:=falseros2 launch ros2_hgr hgr.launch.xmBegin moving your right hand in front of your webcam!
finger_tracking:
Untitled.1.mp4
Uses mediapipe’s hand recognition framework to calculate each of the joint angles of the hand configuration as seen by the RGB camera and publishes it for the motion controller.
go1-gesture-recognition:
3grasp.mp4
Uses a fork of the mediapipe gesture recognition repository to correctly recognize a gesture made by the right hand and assign an id number to it. Depending on the id the hgr_com node publishes the joint angles for corresponding grasping configuration in the robot hand.
allegro_driver:
Untitled.2.mp4
It has 2 nodes, the plan_hand node subscribes to the joint_angles topic and sends it to the move group node to plan a trajectory and then execute it. The allegro_driver node subscribes to the joint_states topic and sends the resulting angles to the robot hand.
Allegro_moveit_config:
Untitled.mp4
Configured the Allegro Hand robot with MoveIt! With the help of MoveIt setup assistant such that its controller can plan and execute valid trajectories for the nearest solution while avoiding self collision and exceeding finger joint limits given goal joint angles.