TTS Objective Metrics 🎯

This repository comprises a compilation of the objective metrics used in several text-to-speech (TTS) papers.

Available Metrics

Metric	Used In
Voicing Decision Error (VDE)	E2E-Prosody, Mellotron
Gross Pitch Error (GPE)	E2E-Prosody, Mellotron
F0 Frame Error (FFE)	E2E-Prosody, Mellotron
Dynamic Time Warping (DTW)	FastSpeech2
Mel Spectral Distortion (MSD)	Wave-Tacotron
Mel Cepstral Distortion (MCD)	E2E-Prosody, Wave-Tacotron
Statistical Moments (STD, SKEW, KURT)	FastSpeech2

Available Pitch Computation

Alogrithm	Proposed In
YIN	(Cheveigné and Kawahara, 2002)
DIO	(Morise, Kawahara, and Katayose, 2009)
PYIN (Testing)	(Mauch and Dixon, 2014)

How to Run

First, clone and enter the repo:

git clone https://github.com/AI-Unicamp/TTS-Objective-Metrics
cd TTS-Objective-Metrics

Install dependencies:

pip install -r requirements.txt

Then, configure the global parameters as you wish in config-> global.py. If not done, all statistics will be computed with the default parameters already set.

The main usage of the repo is to calculate all available metrics for a batch of (ground truth, synthesized) audio pairs (test/eval set). For this make sure you have the (ground_truth, synthesized) audio pairs names matching and in numbered order, each with the same number of digits of the greatest file (eg. if there are 100 files, you shall start with 000.wav, if there are [10,99] you shall start with 00.wav). as in:

📂My Audio
┣ 📂Ground Truths
┃ ┣ 📜00.wav
┃ ┣ 📜01.wav
┃ ┣ ...
┣ 📂Synthesizeds
┃ ┣ 📜00.wav
┃ ┣ 📜01.wav
┃ ┣ ...

Then, choose one pitch computing algorithm and run the following command:

  python -m bin.compute_metrics --gt_folder_path 'My Audio\Ground Truths' --synth_folder_path 'My Audio\Synthesizeds' --pitch_algorithm 'yin'

The result will be saved in a file named metrics.json in the main repo folder: 'TTS Objective Metrics/metrics.json'.

Alternatively, it is possible to calculate a single metric for a pair of (ground truth, synthesized) audio, by choosing an available metric and pitch computation method (yin or dio) with one of the following commands:

# For DTW, FFE, GPE, VDE, moments
python -m metrics.DTW --gt_path 'ground_truth_audio.wav' --synth_path 'synthesized_audio.wav' --pitch_algorithm 'yin'

# For MSD or MCD   
python -m metrics.MSD --gt_path 'path_to_ground_truth_audio.wav' --synth_path 'path_to_synthesized_audio.wav'           

The result will be displayed in the terminal.

Repo Organization

📦TTS Objective Metrics
┣ 📂audio
┃ ┣ 📜helpers.py
┃ ┣ 📜pitch.py
┃ ┣ 📜visuals.py
┣ 📂bin
┃ ┣ 📜compute_metrics.py
┣ 📂config
┃ ┣ 📜global_config.py
┣ 📂metrics
┃ ┣ 📜dists.py
┃ ┣ 📜DTW.py
┃ ┣ 📜FFE.py
┃ ┣ 📜GPE.py
┃ ┣ 📜helpers.py
┃ ┣ 📜MCD.py
┃ ┣ 📜moments.py
┃ ┣ 📜MSD.py
┃ ┣ 📜VDE.py
┣ 📜README.md

How to Contribute

As the repo is still in its infancy, feel free to either open an issue, discussion or send a pull request, or even contact us by e-mail.

Authors

• Leonardo B. de M. M. Marques (leonardoboulitreau@gmail.com)
• Lucas Hideki Ueda (lucashueda@gmail.com)

Github references

• Coqui-AI
• Facebook Fairseq
• NVIDIA Deep Learning Examples
• NVIDIA Mellotron
• MAPS
• Yin

All references are listened on top of the used code itself.