Skip to content
Track facial expressions & get blend shapes, 3D head pose and dense mesh in real-time on iOS, Android, Mac, and PC.
Branch: master
Clone or download

Latest commit

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md Added link to TestFlight demo app Mar 29, 2020

README.md

Facemoji Kit

Bring Facemoji into your app and let users express themselves more comfortably—or their alter ego—in fun, new ways with privacy. Unlike lenses or AR filters that are one-size-fits-all, Facemoji avatars can be personalized.

Facemoji Kit provides an accurate 3D head pose in world space, 51 facial expressions, and optionally a dense 3D mesh of a person’s face. Facemoji Kit uses blend shapes/morph targets (FACS and ARKit-compatible) to animate a 2D or 3D character in ways that follow the user’s facial expressions in real-time.

How it works

RGB Input → NN → Blend shapes weights and pose → Avatar modelling and rendering

Key features

  • Scalable facial tracking
    • A configurable trade-off between accuracy and speed
      • e.g. low-fidelity 2D vs. high-fidelity 3D avatars
  • 1-2MB ML model size
  • 3D reprojection to the input photo/video
  • Supported platforms
    • Unified iOS/Mac OS X Kit with internal platform optimizations
    • Platform suited API and packaging
    • Basic Windows and Android support, full Windows and Android Kits coming soon

Input

RGB image or video

Output

  • FACS and ARKit-compatible blendshapes
  • Gaze tracking
  • Head position in 2D and 3D
  • Head rotation and scale in world coordinates
  • On desktop or high-end phones:
    • Dense mesh (up to 50 000 keypoints)
    • Face segmentation (per pixel)
    • UV and depth estimation

Tracking speed

Higher accuracy

  • 60 FPS on iPhone SE
  • 90 FPS on iPhone X or newer

Lower accuracy

  • 90 FPS on iPhone SE
  • 120 FPS on iPhone X or newer

3D accessories

Facemoji Kit includes all the 3D accessories, models, textures, and shading assets to render Facemoji avatars in your app for your users. There are over 1,000 items, including hair styles, hats, sunglasses, face masks, animated accessories or even face tattoos.

Examples

Demo and download

Get TestFlight demo app or try one of the examples above. If you’re interested in licensing Facemoji Kit, have a question and want to chat, feel free to reach out to us directly via email or join our chat room on Gitter.

You can’t perform that action at this time.