Dual Coder

A Variational Autoencoder (VAE) for log mel spectrogram reconstruction with specialized training for smooth latent space interpolation.

Overview

At its core, this project is a simple VAE that reconstructs log mel spectrograms. We are interested in morphing, so would like to create a smooth and consistent latent space for generating samples from latents interpolated between those for two different sounds in the data set.

The Challenge

We don't have the ground truth for what the interpolations should sound like, but we know that the log mel spectrograms produced by the interpolations should encode "back" to the interpolated latent.

Mathematical Formulation:

L = alpha * l1 + (1-alpha) * l2 → decoder → encoder → (should be) L

Solution

We solve this with a two-phase training approach:

1. Phase 1: Standard VAE training for reconstruction
2. Phase 2: Additional training specifically for interpolated latents to ensure consistency

This dual training approach ensures that interpolated latents in our latent space produce spectrograms that encode back to the same interpolated latent, creating a smooth and morphable latent space.

Usage

Training

Run the two-phase training process:

TwoPhaseTrain.ipynb

Inference & Exploration

Explore your trained model and generate interpolations:

InferencePlayground.ipynb

Requirements

• Python 3.7+
• PyTorch
• NumPy
• Matplotlib
• Jupyter Notebook

Installation

git clone https://github.com/yourusername/dual-coder.git
cd dual-coder
pip install -r requirements.txt

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Version

This is version .00000001 . User at your own risk.