Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Three.js helper for FaceMesh

Demo with textured mask:

Demo with remapped video:

Demo with instanced geometry:

Demo of texture mapping from an image:

Demo of face switching:


How to use

After following the code in the FaceMesh repo, you end up with a face estimation.

First import the class.

import { FaceMeshFaceGeometry } from './face.js';

Create a new geometry helper.

const faceGeometry = new FaceMeshFaceGeometry();

On the update loop, after the model returns some faces:

const faces = await model.estimateFaces(video);
if(faces.length) {

You have to call FaceMeshFaceGeometry::setSize with the width and height of the video to normalise the coordinates so they align with the video source.

That's all there is. You can use faceGeometry as any other BufferGeometry:

const mask = new Mesh(faceGeometry, material);

Mirrored video

This works for the video as it comes. If your source is a webcam, you might want to flip it horizontally.

In that case -besides flipping the video element, by using transform: scaleX(-1), for instance- you'll have to pass a flag to estimateFaces and update:

const faces = await model.estimateFaces(video, false, true);
if(faces.length) {
  faceGeometry.update(faces[0], true);

How to use the video/input as a texture for the face

You can use the input to the FaceMesh model to texture the 3d mesh of the face. Constuct the helper with:

const faceGeometry = new FaceMeshFaceGeometry({useVideoTexture: true});

That will remap the UV coordinates of the geometry to fit the input. The vertex coordinates from the estimation will be projected every frame into the UV space. That means that the UV coordinates for the shader won't work for texture mapping (i.e,, alpha mask, ao map, etc. will be mapped differently and probably wrong).

There seem to be issues with instanced video in macOs Chrome and Safari.

How to use as a 3d mesh, independently of the camera

The range of the vertices is based on the resolution of the input feed, so it changes depending on the chosen video or image input. Construct the FaceGeometry helper with the option normalizeCoords set to true and the mesh will be in a resonable consistent size.

const faceGeometry = new FaceMeshFaceGeometry({normalizeCoords: true});

How to update my threejs camera

FaceMesh data works better with an Orthographic camera. First create a camera:

const camera = new OrthographicCamera(1, 1, 1, 1, -1000, 1000);

and then when the video is ready, or the video dimensions change (width, height), run:

camera.left = -.5 * width;
camera.right = .5 * width; = .5 * height;
camera.bottom = -.5 * height;

Track points in geometry

After faceGeometry.update() you can use faceGeometry.track() to place objects relative to the surface of the face.


It will calculate a triangle defined by the three provided vertex ids, and return a position, a normal, and an orthogonal basis rotation that cane be used to rotate and object along the correct normal of that triangle.

Use this image as a reference for vertex Ids.


FaceMeshFaceGeometry::update(face: FaceEstimation, cameraIsFlipped: boolean): void

Updates the vertices and recalculates normals.

FaceMeshFaceGeometry::setSize(width: number, height: number): void

Sets the internal values to reframe the coordinates.

FaceMeshFaceGeometry::track(id0: number, id1: number, id2: number): { position: Vector3, normal: Vector3, rotation: Matrix4 }

Calculates a triangle defined by vertices id0, id1 and id2, return its center, normal and orthogonal basis.


FaceMeshFaceGeometry for FaceMesh







No releases published


No packages published