Skip to content
This repository contains the code to reproduce the core results from the paper "Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data"
Branch: github_FHVAE
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
conf make it work. Sep 21, 2017
egs fix bugs; add spectrogram to waverform example Jan 30, 2018
img
src minor fix on aurora4 scripts Sep 22, 2017
.gitignore make it work. Sep 21, 2017
LICENSE make it work. Sep 21, 2017
Makefile
NOTICE
README.md
path.sh
requirements.txt

README.md

Factorized Hierarchical Variational Autoencoders

This repository contains the code to reproduce the core results from the paper Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data.

To cite this work, please use

@inproceedings{hsu2017learning,
  title={Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data},
  author={Hsu, Wei-Ning and Zhang, Yu and Glass, James},
  booktitle={Advances in Neural Information Processing Systems},
  year={2017},
}

Dependencies

This project uses Python 2.7.6. Before running the code, you have to install

The former 6 dependencies can be installed using pip by running

pip install -r requirements.txt

The last one requires Kaldi before a specific commit (d1e1e3b). If you don't have Kaldi before that version, you can install both Kaldi and Kaldi-Python by running

make all

Usage

The code structure follows Kaldi's convention. Scripts for each dataset can be found in egs/<dataset> folder. If you have questions, please write an email to wnhsu@csail.mit.edu

TIMIT

To reproduce the experiments for TIMIT, run:

# stage = -1 for data preparation; run only once.
cd egs/timit
./run_fhvae.sh --TIMIT_RAW_DATA <timit_raw_data_dir> --stage -1

Aurora-4

Before running FHVAE experiments, you need to have the following Kaldi data directories of Aurora-4 prepared: <aurora4_kaldi_egs_dir>/data/{train_si84_clean, train_si84_multi, dev_0330, test_eval92}.

To reproduce the experiments for Aurora-4, run:

# stage = -1 for data preparation; run only once.
cd egs/aurora4 
./run_fhvae.sh --AURORA4_KALDI_EGS <aurora4_kaldi_egs_dir> --stage -1

Experimental Results

A. Visualizing Factorization

We show the combination of 6 latent sequence variables and 6 latent segment variables. The latent sequence variable for each column is the same; likewise, the latent segment variable for each row is the same.

TIMIT (alpha = 0)

TIMIT (alpha = 10)

Aurora-4 (alpha = 0)

Aurora-4 (alpha = 10)

B. Utterance-Level S-Vector Modification

We show replacing S-vector of a target utterance with that of a reference utterance to alter the sequence-level attribute, which can be applied to speaker conversion or denoising. The FHVAE model used for following modifications is trained on Aurora-4 with alpha = 0

converting a female speaker to a male speaker

source utterance

reference utterance

S-vector-replaced source utterance

removing background restaurant noise

source utterance

reference utterance

S-vector-replaced source utterance

You can’t perform that action at this time.