Skip to content
Deep Voice: Real-time Neural Text-to-Speech
Branch: master
Clone or download
Latest commit f1ac95a Mar 21, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
deepvoice Engineer G2P; init decoder states to final states Mar 15, 2017
.gitignore Ignore the root data folder only Mar 4, 2017
LICENSE Add license Mar 6, 2017 Rename Keras-2 to Keras Mar 21, 2017

Deep Voice

Join the chat at
Based on the Deep Voice paper.

This repository depends on my Keras fork until it is merged with the official Keras repository.
To install: pip3 install git+
This will override your previously installed Keras version.

Deep Voice is a text-to-speech system based entirely on deep neural networks.

Deep Voice comprises five models:

  • Grapheme-to-phoneme converter.
  • Phoneme Segmentation.
  • Phoneme duration predictor.
  • Frequency predictor.
  • Audio synthesis.



The grapheme-to-phoneme converter converts from written text (e.g English characters) to phonemes (encoded using a phonemic alphabet such as ARPABET).


Based on this architecture but with some changes.

The Grapheme-to-phoneme converter is an encoder-decoder:

  • Encoder: multi-layer, bidirectional encoder, with a gated recurrent unit (GRU) nonlinearity.
  • Decoder: identical to the encoder but unidirectional.

It takes written text as input.


  • Initialization: every decoder layer is initialized to the final hidden state of the corresponding encoder forward layer.
  • Training: the architecture is trained with teacher forcing.
  • Decoding: performed using beam search.


  • Encoder: 3 bidirectional layers with 1024 units each.
  • Decoder: 3 unidirectional layers of the same size as the encoder.
  • Beam Search: width of 5 candidates.
  • Dropout: 0.95 rate after each recurrent layer.

Phoneme Segmentation

  • The phoneme segmentation model locates phoneme boundaries in the voice dataset.
  • Given an audio file and a phoneme-by-phoneme transcription of the audio, the segmentation model identifies where in the audio each phoneme begins and ends.
  • The phoneme segmentation model is trained to output the alignment between a given utterance and a sequence of target phonemes. This task is similar to the problem of aligning speech to written output in speech recognition.


The segmentation model uses the convolutional recurrent neural network based on Deep Speech 2.

The architecture graph

  1. Audio vector.
  2. 20 MFCCs with 10ms stride.
  3. Double 2D convolutions (frequency bins * time).
  4. Triple bidirectional recurrent GRUs.
  5. Softmax.
  6. Output sequence of pairs.



  • Stride: (9, 5).
  • Dropout: 0.95 rate after last convolution.

Recurrent layers

  • Dimensionality: 512 GRU cells for each direction.
  • Dropout: 0.95 rate after the last recurrent layer.


The segmentation model uses the connectionist temporal classification (CTC) loss.

Phoneme Duration + Frequency Predictor


A single architecture is used to jointly predict phoneme duration and time-dependent fundamental frequency.

Phoneme Duration Abstract

The phoneme duration predictor predicts the temporal duration of every phoneme in a phoneme sequence (an utterance).

Frequency Predictor Abstract

The frequency predictor predicts whether a phoneme is voiced. If it is, the model predicts the fundamental frequency (F0) throughout the phoneme’s duration.


  1. A sequence of phonemes with stresses, encoded in one-hot vector.
  2. Double fully-connected layers.
  3. Double unidirectional recurrent layers.
  4. Fully-connected layer.


Double fully-connected layers

  • Dimensionality: 256.
  • Dropout: 0.8 rate after last layer.

Double unidirectional recurrent layers

  • Dimensionality: 128 GRUs.
  • Dropout: 0.8 rate after last layer.

Audio Synthesis


  • Combines the outputs of the grapheme-to-phoneme, phoneme duration, and frequency predictor models.
  • Synthesizes audio at a high sampling rate, corresponding to the desired text.
  • Uses a WaveNet variant which requires less parameters and is faster to train.


The architecture is based on WaveNet but with some changes.

Will be updated soon.

You can’t perform that action at this time.