By default a webcam feedback image is displayed with the face detection frame. The face detection is quite robust to all lighting conditions, but the evaluation of expression can be noisy if the lighting is too directional, too weak or if there is an important backlight. So the webcam feedback image is useful to see the quality of the input video feed.
The computing power of the GPU is also important. If it is powerful, it will process many evaluations per second and the result will be smooth. If you try it on a low end device, even if it works, it may be slow.
Table of contents
- About the tech
- Need more ?
- See also
- face detection and tracking,
- recognize 11 facial expressions,
- face rotation along the 3 axis,
- robust for all lighting conditions,
- mobile friendly,
- example provided using SVG and THREE.js.
/assets/: assets, both for 3D and 2D demonstrations (3D meshes, images),
/demos/: the most interesting: the demos !,
/dist/: heart of the library:
jeelizFaceTransfer.js: main minified script. It gets the webcam video feed, exploit the neural network to detect the face and the expressions and stabilize the result,
jeelizFaceTransferNNC.json: neural network model loaded by the main script,
/doc/: some additionnal documentation,
/helpers/: The outputs of the main script are very raw. It is convenient to use these helpers to animate a 3D model with the THREE.js helper or a SVG file with the SVG helper. All demos use these helpers,
/meshConverter/: only for the THREE.js use. Tool to build the 3D model file including morphs from separate .OBJ files.
<canvas> and audio from the microphone. Then we send them to the server and we encode the video using FFMPEG.
All the following demos are included in this repository, in the
/demos path. You can try them:
If you have not bought a webcam yet, a screenshot video of the Cartman Demo is available here:
This library requires the user's webcam feed through
MediaStream API. So your application should be hosted with a HTTPS server (the certificate can be self-signed). It won't work at all with unsecure HTTP, even locally with some web browsers.
Be careful to enable gzip HTTP/HTTPS compression for JSON and JS files. Indeed, the neuron network JSON in,
/dist/ is quite heavy, but very well compressed with GZIP. You can check the gzip compression of your server here.
The neuron network JSON file is loaded using an ajax
XMLHttpRequest after the user has accepted to share its camera. We proceed this way to avoid to load this quite heavy file if the user refuses to share its webcam or if there is no webcam available. The loading will be faster if you systematically preload the JSON file using a service worker or a simple raw
XMLHttpRequest just after the loading of the HTML page. Then the file will be in the browser cache and will be fast to request.
About the tech
Under the hood
The heart of the lib is
JEEFACETRANSFERAPI. It is implemented by
/dist/jeelizFaceTransfer.js script. It relies on Jeeliz WebGL Deep Learning technology to detect and track the user's face using a deep learning network, and to simultaneously evaluate the expression factors. The accuracy is adaptative: the best is the hardware, the more detections are processed per second. All is done client-side.
The documentation of
JEEFACETRANSFERAPI is included in this repository as a PDF file, /doc/jeefacetransferAPI.pdf. In the main scripts of the demonstration, we never call these methods directly, but always through the helpers. Here is the indices of the morphs returned by this API:
- 0: smileRight → closed mouth smile right
- 1: smileLeft → closed mouth smile left
- 2: eyeBrowLeftDown → eyebrow left frowned
- 3: eyeBrowRightDown → eyebrow right frowned
- 4: eyeBrowLeftUp → eyebrow left up (surprised)
- 5: eyeBrowRightUp → eyebrow right up (surprised)
- 6: mouthOpen → mouth open
- 7: mouthRound → mouth round
- 8: eyeRightClose → close right eye
- 9: eyeLeftClose → close left eye
- 10: mouthNasty → mouth nasty (upper lip raised)
WebGL2is available, it uses
WebGL2and no specific extension is required,
WebGL2is not available but
WebGL1, we require either
WebGL2is not available, and if
WebGL1is not available or neither
OES_HALF_TEXTURE_FLOATare implemented, the user is not compatible.
In all cases, you need to have WebRTC implemented in the web browser, otherwise this library will not be able to get the webcam video feed. The compatibility tables are on, caniuse.com: WebGL1, WebGL2, WebRTC.
If a compatibility error is triggered, please post an issue on this repository. If this is a camera access error, please first retry after closing all applications which could use your device (Skype, Messenger, other browser tabs and windows, ...). Please include:
- a screenshot of webglreport.com - WebGL1 (about your
- a screenshot of webglreport.com - WebGL2 (about your
- the log from the web console,
- the steps to reproduce the bug, and screenshots.
This library works quite everywhere, and it works very well with a high end device like an Iphone X. But if your device is too cheap or too old, it will perform too few evaluations per second and the application will be slow.
We are currently working hard on this project. New neural networks are training and we confident about improving this library. Here are our ways to improve:
- Better emotion detection with a better neural network (improving the structure, the face generator, ...),
- Better tracking stabilization,
- Add a calibration estimation to better take into account the variations of the coefficients between different faces.
JEEFACETRANSFERAPI: All the helpers rely on this API to get the facial morph coefficients. With this documentation you can interface this library with your own 3D or 2D engine. Click here to read the PDF of the specs,
README.md about using the meshConverter: Used in the THREE.JS Fox demo
Articles and tutorials
We will list articles and tutorials about using this library in concrete cases here.
Need more ?
@Jeeliz we have fully developed this library so we can still improve it or fit it to your needs. In particular:
- adapt it to your own 3D/2D engine,
- train a new neural network model adapted for a specific case (take account of new expressions for example),
- use your specific mesh format,
- integrate it better to your workflow,
- manage the video/audio capture and encoding of the weboji.
If you are interested, please contact-us here.
Apache 2.0. This application is free for both commercial and non-commercial use.
We appreciate attribution by including the Jeeliz logo and a link to the Jeeliz website in your application or desktop website. Of course we do not expect a large link to Jeeliz over your face filter, but if you can put the link in the credits/about/help/footer section it would be great.
Jeeliz main face detection and tracking library is called Jeeliz FaceFilter API. It handles multi-face detection, and for each tracked face it provides the rotation angles and the mouth opening factor. It is perfect to build your own Snapchat/MSQRD like face filters running in the browser. It comes with dozen of integration demo, including a face swap.
If you want to detect if the user is looking at the screen or not, Jeeliz Glance Tracker is what you are looking for. It can be useful to play a video only if the user is watching it (and to pause it otherwise). This library needs fewer resources and the neural network file is much lighter.
If are interested by glasses virtual try-on (sunglasses, spectacles, ski masks), you can take a look at Jeeliz VTO widget. It includes a high quality and lightweight 3D engine which implements the following features: deferred shading, PBR, raytraced shadows, normal mapping, ... It also reconstructs the lighting environment around the user (ambient and directional lighting). But the glasses comes from a database hosted in our servers. If you want to add some models, please contact-us.