A Rock-Paper-Scissors game using computer vision and machine learning on Raspberry Pi
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

rps-cv

A Rock-Paper-Scissors game using computer vision and machine learning on Raspberry Pi.

By Julien de la Bruère-Terreault (drgfreeman@tuta.io)

Animated screenshot
Click on image to access video on youtube.

Summary

Project origin

This project results from a challenge my son gave me when I was teaching him the basics of computer programming making a simple text based Rock-Paper-Scissors game in Python. At that time I was starting to experiment with computer vision with a Raspberry Pi and an old USB webcam so my son naively asked me:

"Could you make a Rock-Paper-Scissors game that uses the camera to detect hand gestures?"

I accepted the challenge and about a year and a lot of learning later, I completed the challenge with a functional game.

Overview of the game

The game uses a Raspberry Pi computer and Raspberry Pi camera installed on a 3D printed support with LED strips to achieve consistent images.

The pictures taken by the camera are processed and fed to an image classifier that determines whether the gesture corresponds to "Rock", "Paper" or "Scissors" gestures.

The image classifier uses a Support Vector Machine, a class of machine learning algorithm. The image classifier has been priorly "trained" with a bank of labeled images corresponding to the "Rock", "Paper", "Scissors" gestures captured with the Raspberry Pi camera.

How it works

The image below shows the processing pipeline for the training of the image classifier (top portion) and the prediction of gesture for new images captured by the camera during the game (bottom portion). Click here for full size image. Rock-Paper-Scissors computer vision & machine learning pipeline

Dependencies

The project depends on and has been tested with the following libraries:

  • OpenCV 3.3.0 with bindings for Python 3*
  • Python 3.4+
  • Numpy 1.13.0
  • Scikit-Learn 0.18.2
  • Scikit-Image 0.13.0
  • Pygame 1.9.3
  • Picamera
  • DrGFreeman/PyTools (timer and filter modules)

* Follow this guide for installation of OpenCV on the Raspberry Pi. Install Python libraries within the same virtual environment as OpenCV using the pip install <package_name> command. Picamera is installed by default on Raspbian images.

Hardware:

Camera & lighting setup

Project files

  • capture.py
    This file opens the camera in "capture mode", to capture and label images that will later be used to train the image classifier. The captured images are automatically named and stored in a folder structure.

  • train.py*
    This script reads and processes the training images in preparation for training the image classifier. The processed image data is then used to train the support vector machine image classifier. The trained classifier is stored in the clf.pkl file read by play.py.

  • playgui.py
    This file runs the actual Rock-Paper-Scissors game using the camera and the trained image classifier in a graphical user interface (GUI). Images from each play are captured and added to the image bank, creating additional images to train the classifier.

  • rpsgui.py
    This file defines the RPSGUI class and associated methods to manage the game graphical user interface (GUI).

  • rpsimgproc.py
    This file provides the image processing functions used by the various other Python files.

  • rpsutil.py
    This file provides functions and constants used by the various other Python files.

  • camera.py
    This file defines the Camera class, a wrapper around the picamera library, with specific methods for the project such as white balance calibration.

  • play.py
    This file runs the actual Rock-Paper-Scissors game similarly to playgui.py except the game output is done in the terminal and OpenCV window (no GUI).

* Note that the due to memory limitations on the Raspberry Pi, the train.py script may not run properly on the Raspberry Pi with training sets of more than a few hundred images. Consequently, it is recommended to run these on a more powerful computer. This computer must also have OpenCV, Python 3.4+ and the numpy, scikit-learn and scikit-image Python libraries installed.

Ouput & Screenshots

Training mode

Typical output from train.py (on PC with Intel Core I7-6700 @3.4GHz, 16GB RAM, Anaconda distribution):

(rps-cv) jul@rosalind:~/pi/git/rps-cv$ python train.py
+0.0: Importing libraries
+3.75: Generating image data
Completed processing 1708 images
  rock: 562 images
  paper: 568 images
  scissors: 578 images
+99.51: Generating test set
+99.64: Defining pipeline
+99.64: Defining cross-validation
+99.64: Defining grid search
Grid search parameters:
GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=42, shuffle=True),
       error_score='raise',
       estimator=Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=None,
       random_state=None, svd_solver='auto', tol=0.0, whiten=False)), ('clf', SVC(C=1.0,
       cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3,
       gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True,
       tol=0.001, verbose=False))]),
       fit_params={}, iid=True, n_jobs=4,
       param_grid={'clf__C': array([   1.     ,    3.16228,   10.     ,   31.62278,  100.     ]),
                   'clf__gamma': array([ 0.0001 ,  0.00032,  0.001  ,  0.00316,  0.01   ]),
                   'pca__n_components': [60]},
       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
       scoring='f1_micro', verbose=1)
+99.64: Fitting classifier
Fitting 5 folds for each of 25 candidates, totalling 125 fits
[Parallel(n_jobs=4)]: Done  42 tasks      | elapsed:  2.1min
[Parallel(n_jobs=4)]: Done 125 out of 125 | elapsed:  5.9min finished
Grid search best score: 0.9910406616126809
Grid search best parameters:
  pca__n_components: 60
  clf__C: 10.0
  clf__gamma: 0.00031622776601683794
+458.66: Validating classifier on test set
Classifier f1-score on test set: 0.9922178988326849
Confusion matrix:
[[84  1  0]
 [ 1 84  0]
 [ 0  0 87]]
Classification report:
             precision    recall  f1-score   support

       rock       0.99      0.99      0.99        85
      paper       0.99      0.99      0.99        85
   scissors       1.00      1.00      1.00        87

avg / total       0.99      0.99      0.99       257

+458.72: Writing classifier to clf.pkl
+467.25: Done!

Play mode with Graphical User Interface (playgui.py)

Initial screen:
Initial screen

Computer wins the play:
Computer wins play

Player wins the play:
Player wins play

Tie:
Tie

Game over, player wins the game:
Game over - player wins

Image capture mode (capture.py)

Capture mode

Play mode without GUI (play.py)

Play no GUI