Soundify matches audio clips to video. The tool builds on CLIP to classify scenes (e.g., "bicycle") and retrieves corresponding audio files (e.g., bicycle.wav). Users may layer sounds to create depth and add an additional ambient sound. Soundify is context-aware, being able to adapt retrieved audio files with appropriate panning and volume in a fine-grained manner. The prototype uses Streamlit for its UI.
- Clone the repository.
git clone https://github.com/runwayml/soundify.git
cd soundify- Install package dependencies.
pip install -r requirements.txt-
Download the sound samples, unzip, and save them under the sound directory.
You may add your own sound samples (in .wav format) by adding them under the sound directory and updating main-sounds.txt or ambient-sounds.txt with their filenames (without filename extension).
-
Download the demo video and save it under the root directory.
streamlit run soundify.pyTBD