Skip to content

mAminP/fingerpose-gestures

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

fingerpose-gestures

fingerpose-gestures is a library built on top of Fingerpose classifier for hand landmarks detected by TensorFlow.js' handpose model. It can detect hand gestures inside a webcam source picture.

How it works

Gesture detection works in 3 steps:

  1. Detect the hand landmarks inside the video picture
  2. Estimating the direction and curl of each individual finger
  3. Comparing the result to a set of gesture description

Step (1) is performed by TensorFlow's "handpose", Step (2) and (3) are handled by fingerpose library.

Installation

Install the module via NPM:

npm i --save fingerpose-gestures

Usage

A fully working example can be found inside the demo folder. The basic steps are outlined below:

Include "handpose","fingerpose" and this library

import * as tf from "@tensorflow/tfjs";
import * as handpose from "@tensorflow-models/handpose";
import "@tensorflow/tfjs-backend-webgl";
import * as fp from "fingerpose";
import * as fpg from "fingerpose-gestures";

Configure the gesture recognizer with known gestures

const GE = new fp.GestureEstimator([
    
    // add "✌🏻" and "👍" as sample gestures
    fp.Gestures.VictoryGesture,
    fp.Gestures.ThumbsUpGesture,
    
    // add other gestures by fingerpose-gestures 
    fpg.Gestures.thumbsDownGesture,
    fpg.Gestures.fingerSplayedGesture,
    fpg.Gestures.raisedHandGesture
    
    // ... and more
]);

Use "handpose" to estimate the landmarks

const model = await handpose.load();
const predictions = await model.estimateHands(video, true);

Estimate the gestures

// using a minimum confidence of 7.5 (out of 10)
const estimatedGestures = GE.estimate(predictions.landmarks, 7.5);

The result is an object containing possible gestures and their confidence, for example:

{
    "poseData": [ ... ],
    "gestures": [
        { "name": "thumbs_up", "confidence": 9.25 },
        { ... }
    ]
}

Gestures

name emoji path
thumbs_up 👍 fpg.Gestures.thumbsUpGesture
victory fpg.Gestures.victoryGesture
thumbs_down 👎 fpg.Gestures.thumbsDownGesture
finger_splayed 🖐 fpg.Gestures.fingerSplayedGesture
raised_hand fpg.Gestures.raisedHandGesture
pinching 🤏 fpg.Gestures.pinchingGesture
ok 👌 fpg.Gestures.okGesture
fist fpg.Gestures.fistGesture

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published