Generating MFA aligments

[Fhrozen]

@sw005320 @kan-bayashi @kamo-naoyuki

I prepared this draft based on PRs: #4557 #4801.
It is almost completed for a general support on different data sets.

I only tested on LJSpeech, so some parts are hard coded (stage 1 of .sh script)
In this case the code trains from scratch the mfa's models.
I had some issues with the original PRs and some word did not obtained a phoneme list, instead a spn token.

Training the models, allows to use espnet2-based g2p and reduces the spns to (1) or none (I cannot remember but still got a very few. But using espnet2's g2p then allow to generate the phoneme list and the py script generate a fixated duration for the phone list.

Some samples generated from a trained Fastspeech 2 model + hifigan is located at: https://1drv.ms/u/s!AliZ3I0uDW8HhKB14mq0Sumx_vSbpg?e=IlmQ6T

Let me know if you have any comment.

[iamanigeeit]

Hi @Fhrozen, i remember that spn comes from OOV words, which is why i added the steps to generate OOVs + add them into the MFA dictionaries in my proposed mfa.sh script.

Training MFA models from scratch is definitely another option.

[sw005320]

Thanks a lot!
Should we first check and merge #4801 and then move to this one?
I think #4801 becomes in a good shape now, and it will be merged soon.

[kan-bayashi]

@Fhrozen I just merged #4801. If your PR is alternative or updated version of #4801, please refactor or update scripts added in #4801.

[Fhrozen]

@kan-bayashi I am wondering which option would be better. This PR includes mfa training and It would cover others recipes such as VCTK, so the mfa option should be keep separated as optional (described in the TTS readme) or would become a permanent option in all the tts recipes?

[kan-bayashi]

the mfa option should be keep separated as optional (described in the TTS readme)

I like this one.

[Fhrozen]

@kan-bayashi
I am not sure if this suits your taste.

Instead of executing ./run.sh --stop-stage 1 ==> ./local/mfa.sh ==> ./run.sh --stage 2, I added the local/data.sh in the mfa_align.sh file (scrips/utils/prep_data_mfa_align.sh), so it will only requires to run ./local/mfa.sh ==> ./run.sh --stage 2 (Line 99)

Let me know about it, I will try to finish the PR.

[kan-bayashi]

It looks good :)

[Fhrozen]

@kan-bayashi

I completed the code, so please let me know about any comments on the code.
I will be adding the documentation.

I test the code in LJ, VCTK, and tsukuyomi.
The current code supports:

• single and multiple speakers
• any data set
• can be trained or use pre-trained models.

I will be adding a function lather for downloading pre-trained models from HuggingFace (There are some languages that MFA currently does not support).

A frontend was added in the lab generation for Japanese (and probably Korean and Chinese will require).
This is because the MFA does not recognize Japanese characters without spaces, and the dictionary generation never gets completed.

[codecov]

Codecov Report

Merging #4803 (07ee914) into master (a55853b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4803   +/-   ##
=======================================
  Coverage   79.18%   79.18%           
=======================================
  Files         557      557           
  Lines       49279    49279           
=======================================
  Hits        39020    39020           
  Misses      10259    10259           

Flag	Coverage Δ
test_integration_espnet1	66.39% <ø> (ø)
test_integration_espnet2	49.33% <ø> (ø)
test_python	67.99% <ø> (ø)
test_utils	23.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kan-bayashi]

Sorry for the late review.
The code looks great, nice design that allows various recipe to use MFA.
And I'm glad to hear that Japanese case also works.

Once you complete this PR, let us merge it.

[Fhrozen]

Finish adding some fixes related to the frame generation found due to floating point division.
Documentation was also added.

[Fhrozen]

@kan-bayashi
After testing, it should support any language that the ESPnet front-end supports.

In case of english, I tested with VCTK also and some samples are here
vctk.zip

The files were generating using:

• MFA trained model (cleaner: tacotron, g2p_model: espeak_ng_english_us_vits),
• ProDiff + X-vector + GST model, and
• HifiGan Vocder

For Japanese, I tested with JVS multispeaker + Tsukuyomi
jvs.zip

• MFA trained model (cleaner: jaconv, g2p_model: pyopenjtalk_prosody),
• ProDiff + X-vector + GST model, and
• HifiGan Vocder (vctk)

The issue with the Japanese is the vocoder (I suppose), so I am training a large set for training a vocoder a test again.
Meanwhile, this PR is already finished.
Let me know if there is any fix it is required.

[iamanigeeit]

Hi @Fhrozen -- thanks for integrating the MFA scripts.

I wonder about --g2p_model espeak_ng_english_us_vits in run_mfa.sh. It seems MFA uses a custom IPA symbol set, not just having more symbols but some are different as well (aw vs aʊ̯).

Also, the readme says:

./scripts/mfa.sh --split_sets "train_set dev_set test_set" \
    --stage 1 \
    --stop-stage 2 \
    --train true --nj 36 --g2p_model espeak_ng_english_vits

I think we're supposed to run all (5) stages right?

[Fhrozen]

In case of using espeak_ng_english_us_vits (which is one the choices of the ESPnet frontend), you need to setup --train true and yes, it is required to run all stages to generate the transcriptions with the specified g2p.
In the readme, it is only as general reference.

[iamanigeeit]

@kan-bayashi @Fhrozen There seems to be a naming conflict in the LibriTTS corpus because MFA expects .lab files to be plain text transcriptions, but LibriTTSLabel is in <start> <end> <phone> format. This could potentially impact mfa.sh and trim_silence.py. I am running mfa.sh on LibriTTS to see if the alignment failures are still there because i suspect it's due to OOVs. If it works, i can help to update LibriTTSLabel.

[iamanigeeit]

Also, in mfa.sh, there are two different g2p_model:

1. ESPnet g2p model name for mfa_format.py
2. MFA g2p model for the mfa commands

mfa.sh will complain that either MFA or ESPnet doesn't have this g2p model name.

[Fhrozen]

@iamanigeeit I do not understand the current issue.

There seems to be a naming conflict in the LibriTTS corpus because MFA expects .lab files to be plain text transcriptions, but LibriTTSLabel is in format.

As further I see, it uses normalized text to generate the annotations:

espnet/egs/libritts/tts1/local/data_prep.sh

Lines 57 to 58 in aa88f3a

	txt=$(cat $(echo $wav_file | sed -e "s/\.wav$/.normalized.txt/"))
	echo "$id $txt" >>$trans

espnet/egs2/libritts/tts1/local/data.sh

Lines 78 to 81 in aa88f3a

	cd "${db_root}/LibriTTS/${name}"
	find . -follow -name "*.normalized.txt" -print0 \
	| tar c --null -T - -f - | tar xf - -C "${cwd}/data/local/${name}"
	cd "${cwd}"

The lab files are employed for phoneme alignment I supposed.

Also, in mfa.sh, there are two different g2p_model:

ESPnet g2p model name for mfa_format.py
MFA g2p model for the mfa commands
mfa.sh will complain that either MFA or ESPnet doesn't have this g2p model name.

The setup is for using pretrained espnet mfa models.
There is no issue about that.
if you espnet_ as prefix, the program will automatically use a espnet pretrained mfa model instead of download it from the MFA official page.

I will update this soon for including mfa models stored in Huggingface.

[iamanigeeit]

There seems to be a naming conflict in the LibriTTS corpus because MFA expects .lab files to be plain text transcriptions, but LibriTTSLabel is in <start> <end> <phone> format.

I think this is OK since they are in separate folders (MFA .lab in data/local/mfa/corpus/xx and LibriTTS .lab in downloads/LibriTTS/xx/xx). I was confused because they have the same name.

The setup is for using pretrained espnet mfa models.

I see what you mean now, thanks. If train=true then g2p_model is an existing ESPnet G2P that we use to train MFA G2P. If train=false then g2p_model is a MFA G2P model.