We were both really interested in using VR to create experiences that not everyone has access to. Synthesia, which is the association of sound with color, is one of those experiences. We created an environment where those who don't have synesthesia can surround themselves with the "color" of sound.
We used Unity to simulate this experience. Within the VR space, the color of your surrounding changes depending on the frequency of the real time input to the scene (using FFTs). Within your environment we have a particle system that works similar to that of oscilloscope music. The left and right channels of the audio feed in to the x and y axis.
The z axis represents the time so the particles move further away from you in space as they get "older." We thought it would be cool to see both the frequency and the amplitude in your environment. The entire scene is generated through audio input.
We went through many iterations for this project. Initially we just wanted to create a color changing environment in VR to simulate synesthesia. Then, after discovering oscilloscope music, we were just going to create an oscilloscope music experience in VR.
Parallely, we were experimenting with different audio inputs. We started with pre-generated oscilloscope songs. Working through all the bugs, we were able to get real-time microphone input with a color changing environment and oscilloscope music like signal.
.JPG)
Finally, a video of the experience can be found at the following link: (https://vimeo.com/217347697)
