Skip to content

Pronunciation correction in vector quantized PPG representation space

License

Notifications You must be signed in to change notification settings

warisqr007/vq-bnf-translator

Repository files navigation

Pronunciation correction in vector quantized PPG representation space

Work in progress: Inspired from this paper Zero-Shot Foreign Accent Conversion without a Native Reference

This is the translator module as shown in the above paper.

Installation

  • Install ffmpeg.
  • Install Kaldi
  • Install PyKaldi
  • Install packages using environment.yml file.
  • Download pretrained TDNN-F model, extract it, and set PRETRAIN_ROOT in kaldi_scripts/extract_features_kaldi.sh to the pretrained model directory.

Dataset

  • Acoustic Model: LibriSpeech. Download pretrained TDNN-F acoustic model here.
    • You also need to set KALDI_ROOT and PRETRAIN_ROOT in kaldi_scripts/extract_features_kaldi.sh accordingly.
  • Vector Quantization: [ARCTIC and L2-ARCTIC, see here for detailed training process.
  • Translator seq2seq (i.e., Seq2seq model): ARCTIC and L2-ARCTIC. Please see here for a merged version. All the pretrained the models are available (To be updated) here

Directory layout (Format your dataset to match below)

datatset_root
├── speaker 1
├── speaker 2 
│   ├── wav          # contains all the wav files from speaker 2
│   └── kaldi        # Kaldi files (auto-generated after running kaldi-scripts
.
.
└── speaker N

Quick Start

See the inference script

Training

  • Use Kaldi to extract BNF for individual speakers (Do it for all speakers)
./kaldi_scripts/extract_features_kaldi.sh /path/to/speaker
  • Preprocessing
python preprocess_bnfs.py path/to/dataset
python make_data.py #Edit the file to specify dataset path
  • Vector Quantize the BNFs see here

  • Setting Training params See conf/

  • Training Model

./train.sh
  • Synthesizer Code and Training see here