Music For Memories
Music For Memories is an initial exploration into generative audio for video. The goal is to create a warm and nostalgic soundtrack that can accompany any piece of film. The program analyzes the color values of randomly sampled points along with recently played notes to determine which note from a scale to play next. Due to the nature of using random samples, each viewing of the same video will generate a different soundtrack, allowing for a unique experience every time.
This project was created for the final showcase of Gray Area's 10 week long Creative Code Immersive during Winter 2017. It was displayed in Gray Area's Grand Theater alongside the final project of my peers in the winter cohort. Music For Memories, in its current form, accompanied a compilation of found footage of old family hiking trips captured on 16mm film from the Internet Archive courtesy of David Ross Brower. The video looped for over three hours with the program generating sounds the entire time.
The soundtrack is algorithmically generated by following a set of four steps:
- Sample - take a random point on the current frame of the video and extract its color values.
- Map - seven colors are mapped to the seven notes of a scale. Find which of the seven colors is closest to the sampled color by calculating the average "distance" using RGB values.
- Repetition Detection - look at the notes that have been recently played. If the note from the previous step is the same as the last note played, add a chance to change the note to be one step up or down the scale.
- Chord Variation - adds a chance to play a chord with note selected from the previous steps as the root note.
All sounds are created from samples courtesy of the University of Iowa Electronic Music Studios.
- 11 piano notes on the B flat major third scale. Notes combined in Audacity to create chords.
- 7 flute notes on the B flat major fourth scale stretched to around 30 seconds in length using the Paulstretch python script.
Music For Memories is a processing sketch which requires a video file to create a soundtrack. The Processing video packaged is used to handle video play back and the Minim package is used to trigger audio samples.
Runs in fullscreen mode with the cursor disabled for better presentation.
- IsPlaying() not working on the backing tracks -- probably caused by media event tracking video