Skip to content
Recognize Rock, Scissors and Paper from video captured by camera
Branch: master
Clone or download
Latest commit e1ce534 Mar 29, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
Gesture_Recognition Put blank line back May 20, 2016
.gitattributes change attributes, make text file type text Oct 2, 2015
.gitignore get image from camera Mar 15, 2015
Gesture_Recognition.sln rename project Oct 1, 2015
LICENSE rename project Oct 1, 2015 Readme Revision Mar 29, 2017
appveyor.yml Show file size May 20, 2016


Build status
Recognize Rock, Scissors and Paper from video captured by the camera. Play rock-paper-scissors with your computer!


YouTube Video





You do not need to build the project yourself if you only want to try it. Download the binary EXE from the following link:

The binary above assumes that the computer has at least one webcam, and the front webcam is the default one. If you want to change this setting, you need to modify function CGesture_RecognitionDlg::OnCreate in file Gesture_Recognition/Gesture_RecognitionDlg.cpp

How to use

  • Start the program. Make yourself in the video stream captured by the camera.
  • Do not put your hand in the scene for now. Click the button 'Set Background' which will make the program take a snapshot of the current scene and it will be used for futural background removal.
  • Put your hand in the scene, test if you can see your hand and if your gesture is detected.
  • Start the game by clicking 'Start Game' button.
  • Reset scoreboard by clicking 'Reset Score Board' button.
  • If you see the program detects something in the background on the detection window, you might need to set background again.

Build Environment

Build Steps

  • Download OpenCV 2.4.11 from the link above, unzip all files to C:\opencv (If you don't want to unzip files into C:\opencv, you need to edit /Gesture_Recognition/Gesture_Recognition.vcxproj with some plain text editor)
  • Build the project with MS VS2013
  • Set platform WIN32(x86) and use Release configuration
  • Please use OpenCV with version 2.4.11 exactly
  • If you want to configure project dependency yourself, use staticlib


Please refer to the file Gesture_Recognition/Gesture_Recognition.vcxproj

Implementation Overview

Face Detection

Haar-like features (algorithm implemented in OpenCV).

Skin Color Modeling

Use face color to build skin color model

Apply Skin Color Detection

Detect hand region and remove face

Background Removing

Remove background (noise)

Find Largest Contour

This makes the shape of the hand.

Get Convex Hull and Convexity Defects of Hand

Find Palm Center

Will be used to determine Fingertips

Find Convexity Defects Representing Gaps Between Fingers

Convexity Defects is feature 1.

Find Fingertips

Number of fingertips is feature 2.

Build Decision Tree

With feature 1 & 2

Use History to Avoid Noise

Use a queue to record history

Game and UI Design

Use MFC to implement the UI and randomize the gesture of the computer


  1. Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware, Hui-Shyong Yeo & Byung-Gook Lee & Hyotaek Lim, Dongseo University, Multimedia Tools Application, May 2013
  2. Visual Processing of Rock, Scissors, Paper Game for Human Robot Interaction, Ho-sub Yoon & Su-Young Chi, Division of Intelligent Robot, SICE-ICASE International Joint Conference 2006
  3. Chai D, Ngan KN (1999) Face segmentation using skin-color map in videophone applications. IEEE Trans Circ Syst Video Technol 9(4):551-564
  4. Real-Time Palm Tracking and Hand Gesture Estimation Based on Fore-Arm Contour, Weishao Chen, National Taiwan University of Science and Technology, July 2011.
  5. Vision-Based Finger Action Recognition by Angle Detection and Contour Analysis, Daeho Lee & SeungGwan Lee, ETRI Journal, vol. 33, no. 3, June. 2011, pp. 415-422.
  6. Vision based gesture recognition system with single camera, Wei Du & Hua Li, CAD Lab, Institute of Computing Technology, Chinese Academy of Science, Signal Processing Proceedings, 2000. WCCC-ICSP 2000. 5th International Conference on Volume:2


You must give appropriate credit (link to this repo) if you use (part of) this project IN YOUR WORK. Appropriate copyright info is also required by GNU GPL 3.0

You can’t perform that action at this time.