Skip to content

code for the paper "Differentiable Time-Frequency Scattering on GPU" 🌊

License

Notifications You must be signed in to change notification settings

cyrusvahidi/jtfs-gpu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Differentiable Time-Frequency Scattering on GPU 🌊

source code by

Cyrus Vahidi2
Changhong Wang1, Han Han1 Vincent Lostanlen1
John Muradeli
LS2N/CNRS Centrale Nantes1 Queen Mary University of London2

Paper

Accompanying webpage 🌐
Kymatio: open-source wavelet scattering in Python β€πŸ’»

Many thanks to all open-source contributors to Kymatio and its dependencies.

This repository contains code to replicate the paper "Differentiable Time-Frequency Scattering on GPU" (published at DAFx 2022, best paper award).

Time-frequency scattering is available in Kymatio in beta and will be released in v0.4. To use this implementation, install Kymatio from source. To replicate the results in this paper, follow the installation instructions below.

We assess Time-Frequency Scattering in Kymatio for 3 machine listening research applications:

  • unsupervised manifold learning of spectrotemporal modulations
  • hybrid jtfs + convnet supervised musical instrument classification
  • texture resynthesis

How to run

First, install dependencies

Installation

# clone project   
git clone https://github.com/cyrusvahidi/jtfs-gpu

# install project   
cd jtfs-gpu
pip install -e .
pip install -r requirements.txt

# install kymatio from source
cd kymatio
python setup.py develop
  • The JTFS algorithm source code to replicate the paper can be found here
  • The latest version of JTFS can be installed directly from the Kymatio source

ConvNet Classifier

Download Medley-solos-DB

Extract Scattering Features

python scripts/process_msdb_features.py --data_dir <your_msdb_dir> --feature <feature_to_extract>

Configure gin

In /scripts/gin/config.gin set MSDB_DATA_DIR and MSDB_CSV according to the absolute path of your MSDB download.

Run training

python scripts/train_cnn.py

Isomap Visualizations & K-NN Regression

python scripts/isomap.py

see the output in /img

Scale-Rate Visualizations and Resynthesis

cd notebooks
jupyter notebook

See Scale-Rate Visualization.ipynb and Resynthesis results.ipynb

Notebook Guide

Scale-Rate Visualisations

H E L L O

Synthetic amplitude-modulated chirp dataset

  • Factors of variation:
  • $f_c$ carrier frequency
  • $f_m$ amplitude modulation frequency
  • $\gamma$ chirp rate

Manifold Embedding of the Nearest Neighbour Graph

  • MFCCs
  • Time Scattering
  • Time-Frequency Scattering
  • Open-L3
  • Spectrotemporal Receptive Field (STRF)

K-NN regression of synthesizer parameters

2-D CNN classifier

Differentiable Resynthesis

Citation

@article{muradeli2022differentiable,
  title={Differentiable Time-Frequency Scattering in Kymatio},
  author={John Muradeli, Cyrus Vahidi, Changhong Wang, Han Han, Vincent Lostanlen, Mathieu Lagrange, George Fazekas},
  journal={arXiv preprint arXiv:2204.08269},
  year={2022}
}

About

code for the paper "Differentiable Time-Frequency Scattering on GPU" 🌊

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •