Tacotron 2 (without wavenet)

PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions.

This implementation includes distributed and automatic mixed precision support and uses the LJSpeech dataset.

Distributed and Automatic Mixed Precision support relies on NVIDIA's [Apex] and [AMP].

Visit our [website] for audio samples using our published [Tacotron 2] and [WaveGlow] models.

Pre-requisites

1. NVIDIA GPU + CUDA cuDNN

Setup

1. Download and extract the BZNSYP Speech dataset
2. Clone this repo: git clone https://github.com/rgzn-aiyun/tacotron2-melgan.git
3. CD into this repo: cd tacotron2-melgan
4. Update .wav paths: sed -i -- 's,DUMMY,ljs_dataset_folder/wavs,g' filelists/*.txt
   • Alternatively, set load_mel_from_disk=True in hparams.py and update mel-spectrogram paths
5. Install [Apex]
6. Install python requirements or build docker image
   • Install python requirements: pip install -r requirements.txt

Training

1. python train.py --output_directory=outdir --log_directory=logdir
2. (OPTIONAL) tensorboard --logdir=outdir/logdir

Training using a pre-trained model

Training using a pre-trained model can lead to faster convergence
By default, the dataset dependent text embedding layers are [ignored]

1. Download our published [Tacotron 2] model
2. python train.py --output_directory=outdir --log_directory=logdir -c tacotron2_statedict.pt --warm_start

Multi-GPU (distributed) and Automatic Mixed Precision Training

1. python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

Inference demo

1. Download our published [Tacotron 2] model
2. Download our published [WaveGlow] model
3. jupyter notebook --ip=127.0.0.1 --port=31337
4. Load inference.ipynb

N.b. When performing Mel-Spectrogram to Audio synthesis, make sure Tacotron 2 and the Mel decoder were trained on the same mel-spectrogram representation.

Related repos

melgan-cpu Faster than real time Flow-based Generative Network for Speech Synthesis