IMPORTANT

This project is no longer being maintained as Mediapipe has since added Python bindings and handedness detection which works faster and better than the hacks used here. Furthermore, Mediapipe has since improved their handtracking models. This project will remain in it's current state as a study of models from Mediapipe.

Multi Hand Tracker on Python

Python wrapper for Google's Mediapipe Multi-Hand Tracking pipeline. There are 2 predictor classes available. MultiHandTracker which predicts 2D keypoints and MultiHandTracker3D which predicts 3D keypoints. Keypoints generated from MultiHandTracker3D can be fed into is_right_hand to determine handedness (distinguish between right and left hand). is_right_hand is not part of Mediapipe's pipeline but I thought it'll be useful.

This code is built upon Metal Whale's python wrapper for Mediapipe Hand Tracking.

Getting Started

Basic usage, processing a single image for 2D keypoint predictions:

from PIL import Image
import numpy as np
import multi_hand_tracker as mht
import plot_hand

img_path = "./test_pic.jpg"
img = Image.open(img_path)
img = np.array(img)

palm_model_path = "./models/palm_detection_without_custom_op.tflite"
landmark_model_path = "./models/hand_landmark.tflite"
anchors_path = "./data/anchors.csv" 

# Initialise detector
# the independent flag makes the detector process each image independently
detector = mht.MultiHandTracker(palm_model_path, landmark_model_path, anchors_path, independent = True)

# Get predictions
kp_list, box_list = detector(img)

# Plot predictions
plot_hand.plot_img(img, kp_list, box_list)

Basic usage, processing a single image for 3D keypoint predictions and determining handedness:

from PIL import Image
import numpy as np
import multi_hand_tracker as mht
import plot_hand

img_path = "./test_pic.jpg"
img = Image.open(img_path)
img = np.array(img)

palm_model_path = "./models/palm_detection_without_custom_op.tflite"
landmark_model_path = "./models/hand_landmark_3D.tflite"
anchors_path = "./data/anchors.csv" 

# Initialise detector
# independent flag not implemented for MultiHandTracker3D
detector = mht.MultiHandTracker3D(palm_model_path, landmark_model_path, anchors_path)

# Get predictions
kp_list, box_list = detector(img)

# Determine handedness of each prediction
is_right = [mht.is_right_hand(kp) for kp in kp_list]

# Plot predictions
plot_hand.plot_img(img, kp_list, box_list, is_right=is_right)

Requirements

These are required to use the HandTracker module

numpy
opencv
tensorflow
shapely
scipy

To use the plotting functions you would need matplotlib

Results

Predictions from MultiHandTracker:

Video from Signing Savvy.

Predictions from MultiHandTracker3D:

Video from Signing Savvy.

Not very good when the hands occlude each other.

Acknowledgments

This work is a study of models developed by Google and distributed as a part of the Mediapipe framework.
@metalwhale for the Python Wrapper for single hand tracking and for removing custom operations for the Palm Detections model.