Tones is an interactive application that creates a synesthetic experience for the user. The user is invited to use their keyboard to create different sounds, resembling how they might use a piano. Once sound is produced, an associated color will appear on the screen: warmer colors for higher notes, and cooler colors for lower notes. The colors will dance on the screen as the user plays.
Bonus features include options for the user to record what they've created, as well as choosing between different color schemes and patterns.
With tones, users will be able to:
- interact with their keyboard to create sounds
- use the space key to act as a sustain pedal
In addition, this project will include:
- instructions on how to interact with the page
- a production README
This project implements:
- Canvas API to display shapes and colors
- Web Audio API to create sounds
- Webpack to bundle the Javascript code
Users are able to use their computer keyboard to create sounds. With each keyboard press, the user will hear a sound similar to what a piano keyboard or synthesizer would produce. In addition to these sounds, the user will see shapes of different colors appear on their screen: each color associated with the frequency of sound produced.
I wanted to create the sounds in the same way that synthesizers are built to create sounds. I researched how sounds are created through modifying frequency, pitch, oscillation, and gain. I was able to incorporate these features through the use of Web Audio API, which provides an AudioContext within which you can modify the way each sound is formed. I was able to find a table that layed out all frequencies for an equal-tempered scale, and decided to omit sharp and flat notes in order to produce a unique user-experience.
I attached event listeners to respond to keydown events - every time a user presses a certain key on their keyboard, a sound would play. This worked for a little while; but after around 6 seconds of pressing keys, something would happen that made the sound stop - the keys wouldn't produce sound for about half a second, only to then start working again.
I discovered that I had been creating too many AudioContext instances. I initially set the AudioContext instance as a property of my Audio class - which resulted in creating a new AudioContext instance each time a key was pressed. In order to fix this, I had to create a single AudioContext instance and access it globally. From there I could do whatever I needed to do with each Audio instance.
- Recording functionailty
- Create new color schemes and patterns to choose from
- Add touch event listeners to allow for interaction on a user's phone