OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
C++ Matlab C HTML C# CMake Other
Latest commit 7620b93 Jul 8, 2018
Permalink
Failed to load latest commit information.
cmake More TBB search cleanup Jun 28, 2018
exe Attempting to resolve issues with BLAS and dlib clashes. Jun 21, 2018
gui Bug fix with output aligned faces and videos if no face is detected i… Jun 29, 2018
imgs Adding images for new wiki entries and some code for packaging releases. Dec 28, 2017
lib Bug fix with output aligned faces and videos if no face is detected i… Jun 29, 2018
matlab_runners Changing AU scripts to avoid a potential race condition with director… May 28, 2018
matlab_version Update readme.txt Jul 9, 2018
model_training Moving to OpenFace 2.0! Loads of changes. May 10, 2018
packages ZeroMQ dependencies. Oct 17, 2017
python_scripts Standalone head pose module with ZeroMQ streaming Apr 17, 2017
samples Moving to OpenFace 2.0! Loads of changes. May 10, 2018
.dockerignore Update README with Docker instructions and ignore samples Jun 2, 2018
.gitignore Cleaning up includes, fixing tbb binary include for 32 bit windows ve… Jun 18, 2018
.travis.yml Bug fix with processing really short sequences with FeatureExtraction… Jun 28, 2018
CMakeLists.txt Getting rid of OpenBLAS compilation warnings. Jun 28, 2018
Copyright.txt Moving to OpenFace 2.0! Loads of changes. May 10, 2018
Dockerfile Update README with Docker instructions and ignore samples Jun 2, 2018
OpenFace-license.txt Auto stash before merge of "develop" and "origin/develop" May 9, 2017
OpenFace.sln Updating dlib and linking to it differently, increasing compilation s… Jun 8, 2018
README.md Quick readme update. Jun 29, 2018
appveyor.yml Dlib update preparation for OSX and Ubuntu. Jun 18, 2018
download_models.ps1 Moving to OpenFace 2.0! Loads of changes. May 10, 2018
download_models.sh Moving to OpenFace 2.0! Loads of changes. May 10, 2018
install.sh Bug fix with processing really short sequences with FeatureExtraction… Jun 28, 2018

README.md

OpenFace 2.0.3: an open source facial behavior analysis toolkit

Build Status Build status

Over the past few years, there has been an increased interest in automatic facial behavior analysis and understanding. We present OpenFace – a tool intended for computer vision and machine learning researchers, affective computing community and people interested in building interactive applications based on facial behavior analysis. OpenFace is the first toolkit capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation with available source code for both running and training the models. The computer vision algorithms which represent the core of OpenFace demonstrate state-of-the-art results in all of the above mentioned tasks. Furthermore, our tool is capable of real-time performance and is able to run from a simple webcam without any specialist hardware.

Multicomp logo

Rainbow logo

OpenFace is an implementation of a number of research papers from the Multicomp group, Language Technologies Institute at the Carnegie Mellon University and Rainbow Group, Computer Laboratory, University of Cambridge. The founder of the project and main developer is Tadas Baltrušaitis.

Special thanks goes to Louis-Philippe Morency and his MultiComp Lab at Carnegie Mellon University for help in writing and testing the code, Erroll Wood for the gaze estimation work, and Amir Zadeh and Yao Chong Lim on work on the CE-CLM model.

WIKI

For instructions of how to install/compile/use the project please see WIKI

Functionality

The system is capable of performing a number of facial analysis tasks:

  • Facial Landmark Detection

Sample facial landmark detection image

  • Facial Landmark and head pose tracking (links to YouTube videos)

Multiple Face Tracking Multiple Face Tracking

  • Facial Action Unit Recognition

  • Gaze tracking (image of it in action)

  • Facial Feature Extraction (aligned faces and HOG features)

Sample aligned face and HOG image

Citation

If you use any of the resources provided on this page in any of your publications we ask you to cite the following work and the work for a relevant submodule you used.

Overall system

OpenFace 2.0: Facial Behavior Analysis Toolkit Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency, IEEE International Conference on Automatic Face and Gesture Recognition, 2018

Facial landmark detection and tracking

Convolutional experts constrained local model for facial landmark detection A. Zadeh, T. Baltrušaitis, and Louis-Philippe Morency. Computer Vision and Pattern Recognition Workshops, 2017

Constrained Local Neural Fields for robust facial landmark detection in the wild Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. in IEEE Int. Conference on Computer Vision Workshops, 300 Faces in-the-Wild Challenge, 2013.

Eye gaze tracking

Rendering of Eyes for Eye-Shape Registration and Gaze Estimation Erroll Wood, Tadas Baltrušaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, and Andreas Bulling in IEEE International. Conference on Computer Vision (ICCV), 2015

Facial Action Unit detection

Cross-dataset learning and person-specific normalisation for automatic Action Unit detection Tadas Baltrušaitis, Marwa Mahmoud, and Peter Robinson in Facial Expression Recognition and Analysis Challenge, IEEE International Conference on Automatic Face and Gesture Recognition, 2015

Commercial license

For inquiries about the commercial licensing of the OpenFace toolkit please visit https://www.flintbox.com/public/project/50632/

Final remarks

I did my best to make sure that the code runs out of the box but there are always issues and I would be grateful for your understanding that this is research code and a research project. If you encounter any problems/bugs/issues please contact me on github or by emailing me at tadyla@gmail.com for any bug reports/questions/suggestions. I prefer questions and bug reports on github as that provides visibility to others who might be encountering same issues or who have the same questions.

Copyright

Copyright can be found in the Copyright.txt

You have to respect boost, TBB, dlib, OpenBLAS, and OpenCV licenses.

Furthermore you have to respect the licenses of the datasets used for model training - https://github.com/TadasBaltrusaitis/OpenFace/wiki/Datasets