Update 22/03/2017 Fixed a bug in mask_analysis.py and almost completed a more robust version of the CNN head pose estimator.
Update 28/10/2016 New package motion_detection.py. Using the classes in this package it is possible to track moving objects through background subtraction. Possible applications of this algorithm are people detection, vehicle detection and tracking [example]
Update 5/10/2016: Working example on how to use CNNs for head pose estimation (for the moment only yaw angle) [code]
Update 20/09/2016: Work in progress. The code provided at the moment does not still implement gaze detection. There is a beta version of the class which implements the CNN head pose estimator of the yaw angle [code]. You can use it loading the variables stored in this [file].
What is deepgaze?
Deepgaze is a library for people detection and tracking which uses Convolutional Neural Networks (CNNs) to estimate the Focus of Attention (FOA) of users. The FOA can be approximately estimated finding the head orientation. This is particularly useful when the eyes are covered, or when the user is too far from the camera to grab the eye region with a good resolution. When the eye region is visible it is possible to estimate the gaze direction, which is much more informative and can give a good indication of the FOA. Deepgaze contains useful packages for:
- Head pose estimation (Perspective-n-Point, Convolutional Neural Networks)
- Face detection (Haar Cascade)
- Skin and color detection (Range detection, Backprojection)
- Motion detection (Frame differencing, MOG, MOG2)
- Motion tracking (Particle filter)
Deepgaze is based on OpenCV and Tensorflow, some of the best libraries in computer vision and machine learning. Deepgaze is an open source project and any contribution is appreciated, feel free to fork the repository and propose integrations.
This library is the result of my recent work which is under revision: Head Pose Estimation in the Wild using Convolutional Neural Networks and Adaptive Gradient Methods
What is a Convolutional Neural Network?
A convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. They have wide applications in image and video recognition, recommender systems and natural language processing [wiki]
To use the libray you have to install:
- Numpy [link]
sudo pip install numpy
- OpenCV [link]
sudo apt-get install libopencv-dev python-opencv
- Tensorflow [link]
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl sudo pip install --upgrade $TF_BINARY_URL
Some examples may require additional libraries:
- dlib [link]
Download the repository from [here] or clone it using git:
git clone https://github.com/mpatacchiola/deepgaze.git
To install the package you have to run the setup.py script (it may require root privileges):
sudo python setup.py install
If you want to track all the installed files you can record the installation process in a text file:
sudo python setup.py install --record record.txt
Done! Now give a look to the examples below.
Head Pose Estimation in images using Convolutional Neural Networks [code]
- Skin detection using the HSV range color detector [code]
- Face detection using the HSV range color detector [code]
- Motion detection and tracking using frame differencing on a video streaming [code]
- Motion tracking with multiple backprojection for playing chrome's dinosaur game [blog] [code] [video]
The example "head pose estimation using Perspective-n-Point" is partially based on the C++ version you can find here, and on the workshop "Developing an attention system for a social robot" which was part of the 2nd International Summer School on Social Human-Robot Interaction.