Ear training game using machine learning (ML/AI) models in the browser
🏆 This project is a winning entry to AIPilipinas (AI Philippines) TensorFlow Project Showcase 2020-2021!
Audiate, stylized audiate, is a play-what-you-hear type of game wherein you are tasked to repeat the sequence of notes that the app plays, using the instrument of your choice. It aims to test (& train through practice) one's musical memory and relative pitch.
This app runs in your web browser and you can interact with it using the sound of your instrument, fed through your device's microphone. The captured audio is fed to a pitch transcription model running in the browser which transcribes the notes you played, and then checked if matches with the reference notes that the app instructed you to play. Using the transcription model eliminates the need for music interfaces such as MIDI as it can extract the pitch information from plain audio.
Watch this for a simple demo :)
App has been tested to work well in Chrome in Ubuntu. In some tests, Windows 10 and Android Q (yes, on phone) can work too. Note also that the pitch transcription model may not transcribe pitch with 100% accuracy, especially on noisy environments.
You may try the app in the GitHub pages or on your local machine by cloning this repository and serving the app, for example, via python:
$ python -m http.server 5000
This app makes use of a pre-trained pitch tracker model, CREPE. With only 2MB model size, it can run without hogging much of your device's resources.
An earlier version of audiate, called pianotize, is also available in another branch. It makes use of the Onsets & Frames piano transcription model but is a lot slower and less accurate for this type of application.
- TensorFlow.js
- Magenta.js (old model)
- Tone.js
- Tonal.js
- P5.js