Face DJ

Face DJ is a face-controlled DJ web application that works in Chrome or Firefox.

The app uses the Tensorflow Facemesh library to accurately detect facial movements and use these data to control music.

I created this app to explore how face-detection technology can be used for controlling/producing music and to provide a more immersive musical experience.

Here's the Live Demo on Glitch.

You need a webcam, headphones, and a microphone to fully use this app. Please try the app in the latest version of Chrome or Firefox. (Safari is currently not supported)

Features

• Moving your head by X, Y position changes the song pattern.
• Moving your head by Z position changes the volume.
• Tilting the head based on the Y-axis controls the panning.
• Tilting the head based on the X-axis controls the filter cutoff.
• Opening/closing your mouth controls the filter resonance.
• Saying something to the microphone plays a vocoder.

Setup

1. Installation of node.js is required. Follow this guide to install it.
2. Run the following commands in the Terminal.

git clone https://github.com/cuinjune/face-dj.git
cd face-dj
npm install dependencies
npm start

3. Open your web browser and navigate to http://localhost:3000

Tools & Assets used

Tensorflow Facemesh library, Three.js, Pure Data, libpd, SDL, Emscripten, Node.js, Express.js, Royalty Free Music from Bensound, Hifi Stereo 3D Model from Google Poly.

Reporting bugs

Please post an issue if you face any problem (e.g. no music playing after clicking the “Play Music” button or speakers not moving) using the app.

Author

• Zack Lee: MPS Candidate at NYU ITP.