Permalink
Browse files

Add sphere demo docs.

  • Loading branch information...
1 parent 8bc8227 commit bb64d6917e3e9ccb5c3cbde3469a44849054c941 @bamos bamos committed Sep 15, 2016
Showing with 101 additions and 4 deletions.
  1. +5 −1 docs/demo-1-web.md
  2. +5 −1 docs/demo-2-comparison.md
  3. +6 −1 docs/demo-3-classifier.md
  4. +79 −0 docs/demo-4-sphere.md
  5. +4 −0 docs/index.md
  6. +2 −1 mkdocs.yml
View
@@ -1,4 +1,8 @@
-# Demo 1: Real-Time Web Demo
+## Demo 1: Real-Time Web Demo
+Released by [Brandon Amos](http://bamos.github.io) on 2015-10-13.
+
+---
+
See [our YouTube video](https://www.youtube.com/watch?v=LZJOTRkjZA4)
of using this in a real-time web application
for face recognition.
@@ -1,4 +1,8 @@
-# Demo 2: Comparing two images
+## Demo 2: Comparing two images
+Released by [Brandon Amos](http://bamos.github.io) on 2015-10-13.
+
+---
+
The [comparison demo](https://github.com/cmusatyalab/openface/blob/master/demos/compare.py) outputs the predicted similarity
score of two faces by computing the squared L2 distance between
their representations.
@@ -1,4 +1,9 @@
-# Demo 3: Training a Classifier
+## Demo 3: Training a Classifier
+
+Released by [Brandon Amos](http://bamos.github.io) on 2015-10-13.
+
+---
+
OpenFace's core provides a feature extraction method to
obtain a low-dimensional representation of any face.
[demos/classifier.py](https://github.com/cmusatyalab/openface/blob/master/demos/classifier.py)
View
@@ -0,0 +1,79 @@
+## Demo 4: Real-Time Face Embedding Visualization
+Released by [Brandon Amos](http://bamos.github.io) and
+[Gabriel Farina](https://github.com/gabrfarina) on 2016-09-12.
+
+---
+
+![](https://raw.githubusercontent.com/cmusatyalab/openface/master/images/sphere-demo/demo.gif)
+
+<center>
+![](https://raw.githubusercontent.com/cmusatyalab/openface/master/images/sphere-demo/exhibit-amos.png)
+</center>
+
+We had a great opportunity
+(*thanks to Jan Harkes, Alison Langmead, and Aaron Henderson*)
+to present a short OpenFace demo
+in the [Data (after)Lives art exhibit](https://uag.pitt.edu/Detail/occurrences/370)
+at the University of Pittsburgh, which is live from Sept 8, 2016 to Oct 14, 2016
+and investigates the relationship between the human notions of self and
+technical alternative, externalized, and malleable representations of identity.
+We have released the source code behind this demo in our main
+GitHub repository in
+[demos/sphere.py](https://github.com/cmusatyalab/openface/blob/master/demos/sphere.py).
+This exhibit also features [two other art pieces](https://raw.githubusercontent.com/cmusatyalab/openface/master/images/sphere-demo/exhibits-nosenzo.png)
+by [Sam Nosenzo](http://www.pitt.edu/~san76/),
+[Alison Langmead](http://www.haa.pitt.edu/person/alison-langmead/),
+and [Aaron Henderson](http://www.aaronhenderson.com/) that use OpenFace.
+
+### How this is implemented
+
+This is a short description of our implementation in
+[demos/sphere.py](https://github.com/cmusatyalab/openface/blob/master/demos/sphere.py),
+which is only ~300 lines of code.
+
+For a brief intro to OpenFace, we provide face recognition with
+a deep neural network that embed faces on a sphere.
+(See [our tech report](http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-118.pdf)
+for a more detailed intro to how OpenFace works.)
+Faces are often embedded onto a 128-dimensional sphere.
+For this demo, we re-trained a neural network to embed faces onto a
+3-dimensional sphere that we show in real-time on top of a webcam feed.
+The 3-dimensional embedding doesn't have the same accuracy as the
+128-dimensional embedding, but it's sufficient to illustrate how
+the embedding space distinguishes between different people.
+
+In this demo:
+
++ We first use [OpenCV](http://opencv.org/) to get, process, and display
+ a video feed from the camera.
++ The detected faces and embeddings for every face can be easily obtained with
+ [dlib](http://blog.dlib.net/) and OpenFace with
+ [a few lines of code](http://cmusatyalab.github.io/openface/usage/).
++ The color of the embedding is created by mapping the location of the
+ face in the frame to be a number between 0 and 1 and then using
+ a [matplotlib colormap](http://matplotlib.org/examples/color/colormaps_reference.html).
++ To keep all of the graphics on a single panel, we draw the sphere on
+ top of the same OpenCV buffer as the video.
+ [OpenCV only has 2D drawing primitives](http://docs.opencv.org/2.4/modules/core/doc/drawing_functions.html),
+ so we [isometrically project](https://en.wikipedia.org/wiki/Isometric_projection)
+ the points from the 3D sphere into 2D so we can use OpenCV's 2D drawing primitives.
++ Since the images from the video are noisy, the embeddings will jump around
+ a lot of the sphere if not dampened.
+ We smooth this out with
+ [dlib's object tracker](http://blog.dlib.net/2015/02/dlib-1813-released.html)
+ to track of a face's average (dampened) embedding throughout
+ the video frames.
++ Face detection and recognition cause the 'low' frame rate.
+ The frame rate could be improved by only doing detection and recognition
+ every few frames and using face tracking (which is fast) in between to
+ update the face locations.
+
+### Running on your computer
+
+To run this on your computer:
+
+1. [Set up OpenFace](http://cmusatyalab.github.io/openface/setup/).
+2. Download the 3D model from
+ [here](http://openface-models.storage.cmusatyalab.org/nn4.small2.3d.v1.t7).
+3. Run [demos/sphere.py](https://github.com/cmusatyalab/openface/blob/master/demos/sphere.py)
+ with the `--networkModel` argument pointing to the 3D model.
View
@@ -9,6 +9,10 @@ deep neural networks.*
## News
++ 2016-09-15: We presented OpenFace in the
+ [Data (after)Lives](https://uag.pitt.edu/Detail/occurrences/370) art exhibit
+ at the University of Pittsburgh and have released the code as
+ [Demo 4: Real-time Face Embedding Visualization](demo-4-sphere/).
+ 2016-08-09: [New blog post: (Face) Image Completion with Deep Learning in TensorFlow](http://bamos.github.io/2016/08/09/deep-completion/). ([OpenFace group discussion on it](https://groups.google.com/forum/#!topic/cmu-openface/h7t-URw7zJA))
+ 2016-06-01: [OpenFace tech report released](http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-118.pdf)
+ 2016-01-19: OpenFace 0.2.0 released!
View
@@ -14,6 +14,7 @@ pages:
- Demo 1 - Real-time Web: demo-1-web.md
- Demo 2 - Comparison: demo-2-comparison.md
- Demo 3 - Training a Classifier: demo-3-classifier.md
+ - Demo 4 - Real-time Sphere Visualization: demo-4-sphere.md
- User Guide:
- Usage and API Docs: usage.md
- Setup: setup.md
@@ -22,4 +23,4 @@ pages:
- Models and Accuracies: models-and-accuracies.md
- Training a DNN Model: training-new-models.md
- Visualizations: visualizations.md
-- Release Notes: release-notes.md
+- Release Notes: release-notes.md

0 comments on commit bb64d69

Please sign in to comment.