BandIt: Cinematic Audio Source Separation

Code for "A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation" by Karn N. Watcharasupat, Chih-Wei Wu, Yiwei Ding, Iroro Orife, Aaron J. Hipple, Phillip A. Williams, Scott Kramer, Alexander Lerch, William Wolcott.

Paper

Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.

For Demo

Go here for demo of selected models using the first 10 files from DnR test set. Go here for exhaustive inference on the entire DnR test set for selected models.

For Replication

• Install required dependencies from environment.yaml.
• Obtain DnR dataset from here and MUSDB18-HQ from here.
• Run each dataset's respective proprocess.py.
• python train.py expt/path-to-the-desired-experiment.yaml.
• python test.py expt/path-to-the-desired-experiment.yaml --ckpt_path=path/to/checkpoint-from-training.ckpt.

For Inference

• Get the checkpoints from Zenodo
• Get the corresponding yaml config file from expt.
• Put the checkpoint and the yaml config file into the same subfolder. Rename the config file hparams.yaml.

python inference.py inference \
  --ckpt_path=path/to/checkpoint.ckpt \
  --file_path=path/to/file.wav \
  --model_name=model_id

or

python inference.py inference_multiple \
  --ckpt_path=path/to/checkpoint.ckpt \
  --file_glob=path/to/glob/*.wav \
  --model_name=model_id

Complexity Benchmark

• Intel Core i9-11900K CPU + NVIDIA GeForce RTX 3090 GPU.

Model	Band	GFlops	Params (M)	Peak Memory (MB)	Batch per second (GPU)	Batch per second (CPU)	GPU speedup
BSRNN-GRU8 (per stem)	Vocals V7	238.2	15.8	416.2	12.35	0.61	20.2
BSRNN-LSTM12 (per stem)	Vocals V7	462.2	25.8	505.4	7.99	0.6	13.4
BandIt	Bark 48	290.6	64.5	643	10.22	0.39	26.1
BandIt	ERB 48	274.2	32.6	519.5	10.31	0.41	25
BandIt	Mel 48	274.3	32.8	519.3	10.15	0.38	26.6
BandIt	Music 48	274.7	33.5	524.2	10.22	0.43	23.5
BandIt	TriBark 48	274.2	32.7	519.9	10.3	0.4	25.5
BandIt	Bark 64	387.6	82.6	828.5	8.64	0.4	21.9
BandIt	ERB 64	363.5	36	649.1	8.68	0.42	20.7
BandIt	Mel 64	363.6	36.1	648.9	8.71	0.32	27.2
BandIt	Music 64	364.1	37	653	8.69	0.31	27.7
BandIt	TriBark 64	363.5	36	648.7	8.68	0.42	20.6
BandIt	Vocals V7	243.2	25.7	454	11.34	0.6	18.8
Hybrid Demucs		85	83.6	552.5	17.04	1.1	15.5
Open-Unmix		5.7	22.1	187.8	52.5	20.77	2.5

Citation

@ARTICLE{10342812,
  author={Watcharasupat, Karn N. and Wu, Chih-Wei and Ding, Yiwei and Orife, Iroro and Hipple, Aaron J. and Williams, Phillip A. and Kramer, Scott and Lerch, Alexander and Wolcott, William},
  journal={IEEE Open Journal of Signal Processing}, 
  title={A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation}, 
  year={2024},
  volume={5},
  number={},
  pages={73-81},
  doi={10.1109/OJSP.2023.3339428}}