EMSYNC: Video-to-Music Generation

A system for generating emotionally and temporally synchronized music from video content.

Paper | Samples and Colab demo | Video presentation

If you use this work in your research, please consider citing our work:

S. Sulun, P. Viana, and M. E. P. Davies, “Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries,” IEEE Transactions on Multimedia, 2026.

Installation

Install required Python libraries:

pip install -r requirements.txt

Quick Start

Inference

Generate music for your video using pre-trained models (automatically downloads required models):

python inference.py --input_path sample.mp4 --output_path output/sample_output.mp4

Training Your Own Model

1. Download training dataset

Download and extract the Lakh Pianoroll 5 full (LPD-5-full) dataset:

• Homepage: https://hermandong.com/lakh-pianoroll-dataset/dataset
• Direct download: https://ucsdcloud-my.sharepoint.com/:u:/r/personal/h3dong_ucsd_edu/Documents/data/lpd/lpd_5/lpd_5_full.tar.gz?csf=1&web=1&e=sPANiy

# Extract the downloaded dataset
tar -xzf lpd_5_full.tar.gz

2. Preprocess data

python -m midi.src.data.preprocess --input_dir lpd_5/lpd_5_full --output_dir lpd_5/processed

3. Train model

Train the music generation model (check config.py for hyperparameters):

python -m midi.src.train --data_dir lpd_5/processed

Evaluation

Dataset setup

Navigate to evaluation directory and download evaluation datasets:

cd evaluation/data

Download EmoMV dataset

./download_emomv.sh

Download and filter Ads dataset

python download_ads.py

Batch inference

Run inference on all evaluation datasets:

cd ..
./run_inference_on_datasets.sh

Objective evaluation metrics

Audio-video alignment (AV-Align)

Measure synchronization between generated music and video content:

python get_av_alignment.py

KL-divergence

Evaluate emotional consistency between video and generated music:

python get_kl_divergence.py

Subjective evaluation

Analyze survey results:

python analyze_surveys.py