A video demo of the application is available at this link.
The environments we daily deal with constantly provide us with sounds, melodies and noises generating different soundscapes, but have you ever wondered how a song you like would sound in those contexts? Here is the the goal of Soundscape, a Web application meant to make you feel the sounds that surround you in a different way, creating a special bond between music and nature.
- Select, upload or record a song
- Choose the soundscape you like
- Hear the result!
- The available soundscapes are Srping Mountain, Sea Life and Natural Pond
- The song must contain at least some vocals or a bass line or a drum pattern or all of them to properly work.
The whole process is made of three main steps:
The input audio is analysed and divided into four stems (Vocals, Bass, Drums, Accompaniment) using Spleeter by Deezer, all extracted at 16kHz. Then for each stem the RMS energy is computed to check if the track is actually present.
Each available soundscape has two harmonic sounds which can be used to resynthesize the vocal and bass lines, if present, depending on their range. This is done by performing timbre transfer thanks to DDSP by Magenta.
Each harmonic model is trained on a dataset containing between 15 and 20 minutes of the target sound. Due to the difficulty in finding those environmental sounds with a high variability, quality and length, data augmentation was performed making sure to keep the timbre equal through the whole dataset. A preprocessing step of denoising and compressioin was also necessary.
The percussive part is resynthesized trying to match each drum sound with the most similar noise form the chosen soundscape. This is done by using Omnizart by MCTLab. The percussive sound analyisis and classification regard four types of sounds: Hi-hat, Kick, Snare and Clap. For each soundscape a soundfont was created, containing a series of different sounds belonging to it with different envelopes in order to cover as many percussive sounds as possible, then a randomization of similar soundfont sounds is performed to obtain variety.
# Create python virtual environment and activate it:
py -m venv soundscape-venv
.\soundscape-venv\Scripts\activate
# Install dependencies and download required data:
pip install -U pip
pip install cython
pip install numpy
pip install -r requirements.txt
omnizart download-checkpointsSubstitute soundscape-venv\Lib\site-packages\omnizart\drum\labels.py and soundscape-venv\Lib\site-packages\ddsp\colab\colab_utils.py with the provided labels.py and colab_utils.py files.
This application was developed as a project for the "Creative Programming and Computing" course at Politecnico di Milano (MSc in Music and Acoustic Engineering).