[Bug] Gruut espeak inconsistencies makes the training harder.

[erogol]

Describe the bug
Inconsistency b/w gruut with speak phonemes vs phonemizer. Gruut adds additional : between characters. It breaks the pronunciation especially as saying to or to be.

To Reproduce
For the sentence It took me quite a long time to develop a voice, and now that I have it I'm not going to be silent.

Phonemizer with espeak-ng:

echo "It took me quite a long time to develop a voice, and now that I have it I'm not going to be silent." | phonemize -b espeak -l en-gb
ɪt tʊk miː kwaɪt ɐ lɒŋ taɪm tə dɪvɛləp ɐ vɔɪs and naʊ ðat aɪ hav ɪt aɪm nɒt ɡəʊɪŋ təbi saɪlənt 

Gruut:

text2phone(text, LANG, use_espeak_phonemes=True)
'ɪ|t| t|ʊ|k| m|iː| k|w|aɪ|t| ɐ| l|ɔ|ŋ| t|aɪ|m| t|uː| d|ɪ|v|ɛ|l|ə|p| ɐ| v|ɔɪ|s| ,| æ|n|d| n|aʊ| ð|æ|t| aɪ| h|æ|v| ɪ|t| aɪ|m| n|ɑː|t| ɡ|oʊ|ɪ|ŋ| t|uː| b|iː| s|aɪ|l|ə|n|t| .'

Additional context
I see that these inconsistencies make the learning harder for 🐸 TTS models.

In general training, a model with raw chars produces good results faster than phoneme-based training. I assume this is because of such inconsistencies between the phonemizer and gruut.

I am not training a model with use_espeak_phonemes=False and see if it makes any difference.

[synesthesiam]

It looks like this is happening because pronunciations in espeak-ng are highly context dependent:

espeak-ng -v en-us -q --ipa 'to'
 tˈuː

versus

espeak-ng -v en-us -q --ipa 'to develop'
 tə dɪvˈɛləp

When creating my eSpeak lexicons, I get phonemes for each word individually. So the pronunciation for "to" is coming through as tˈuː instead of tə.

For English, I was able to get part-of-speech aware eSpeak pronunciations by prepending a specific word. For example, had <WORD> will trigger the past tense pronunciation of <WORD>:

espeak-ng -v en-us -q --ipa 'wound'
 wˈuːnd

espeak-ng -v en-us -q --ipa 'had wound'
 hæd wˈaʊnd

Similarly, adding another word after <WORD> should trigger the desired pronunciation. The obvious choice is just <WORD> <WORD>, but this will take some more thought because eSpeak tries to be extra "smart" and will happily combine words:

espeak-ng -v en-us -q --ipa 'to to'
 tʊtˈʊ

EDIT: After some more tests, it looks like I can stop this behavior by injecting phonemes for the second word instead:

espeak-ng -v en-us -q --ipa "to [[ x ]]"
 tˈə x

Unless you have another suggestion, I can re-generate the eSpeak lexicons with this method and see if it helps.

[erogol]

Sounds good to me but seemingly the contextual differences will be a problem if not solved by the TTS model.

Do you know how Espeak deals with the context?

Just a side note, training models with use_espeak_phonemes=False work significantly better.

[synesthesiam]

Do you know how Espeak deals with the context?

For part of speech at least, it seems to maintain a set of counters that mean things like "expect perfect tense in the next N words" or "don't expect a verb next".

I haven't found yet where and how it decides to alter the pronunciation of a word just due to the presence of other words, or what triggers the decision to combine two words into one.

Just a side note, training models with use_espeak_phonemes=False work significantly better.

OK, this is good to know. At least there's a viable alternative until I can get the eSpeak problems sorted out.

[synesthesiam]

Small side note: you can actually trigger this behavior in phonemizer as well due to how it handles punctuation preservation!

echo 'To, be.' | phonemize 
tə biː

echo 'To, be.' | phonemize --preserve-punctuation
tuː , biː .

echo 'To be.' | phonemize 
tə biː

The punctuation mark right after "to" causes it to get run through eSpeak alone, triggering its "single word" pronunciation. It's only a corner case here, but makes me wonder what other weird behaviors of eSpeak lurk beneath... 😉

[thorstenMueller]

I do not want to hijack this discussion and am not sure if this has been discussed somewhere else, but i'm confused how Gruut deals with foreign language words in a sentence? In Germany we use some english words in german sentences which are pronounced wrong.

espeak

echo "Ein Song geht mir nicht mehr aus den Ohren." | phonemize -b espeak -l de

[WARNING] 1 utterances containing language switches on lines 1
[WARNING] extra phones may appear in the "de" phoneset
[WARNING] language switch flags have been kept (applying "keep-flags" policy)
aɪn (en)sɒŋ(de) ɡeːt miːɾ nɪçt meːɾ aʊs deːn oːrən 

TTS server with Gruut

It seems that Gruut recognizes it's an english word and is tagging it right, but the spoken audio doesn't sound english.

 > Model input: Ein Song geht mir nicht mehr aus den Ohren.
 > Text splitted to sentences.
['Ein Song geht mir nicht mehr aus den Ohren.']
 > Phonemes: aɪ|n| (en)|s|ɒ|ŋ|(de)| ɡ|eː|t| m|iː|ɾ| n|ɪ|ç|t| m|eː|ɾ| aʊ|s| d|eː|n| oː|r|ə|n| .
 > Processing time: 2.6341404914855957
 > Real-time factor: 0.7461818838291031

I tried following sentence with current Coqui release:

Magst Du den Song der auf der Party lief?

Leading to following phonemes:

Phonemes: m|ɑː|k|s|t| d|uː| d|eː|n| (en)|s|ɒ|ŋ|(de)| d|ɛ|ɾ| aʊ|f| d|ɛ|ɾ| (en)|p|ɑː|t|i|(de)| l|iː|f| ?

Here's the spoken output: https://sndup.net/4jcy

It sounds a little bit as the language tags will be spoken too ;-)
Could that be the case?

[synesthesiam]

Thank you for this, Thorsten. Just when I think I've figured out all of eSpeak's quirks, it hits me something new 🤦‍♂️

There does not appear to be any option for disabling the "language switching" flags that eSpeak so "helpfully" inserts into the phoneme output. I had assumed they were opt-in, but phonemizer must have been manually removing them with a regex or something.

I will rebuild the eSpeak lexicons with these flags removed. Thank you!

[synesthesiam]

I've pushed up a new version of gruut (1.3.1) with updated eSpeak lexicons and g2p models. I can create a PR for the new version if that's OK with you, @erogol

The updated lexicons are produced by doubling each word like this:

echo 'a,a' | espeak-ng -v en-us -q --ipa
 ɐ ˈeɪ

and then taking the phonemes from the first word (ɐ in this case).

This works for the original bug @erogol reported, but produces a new "problem" with other words:

echo 'it,it' | espeak-ng -v en-us -q --ipa
 ɪɾ ˈɪt

eSpeak replaces t with the flap ɾ presumably because the next word begins with a vowel. I manually changed this in gruut's lexicon, but other words may behave similarly.

If I can understand eSpeak's rules better, I should be able to create context-sensitive pronunciations for gruut that follow these rules. Right now, part of speech is the only feature used to disambiguate multiple pronunciations, but others (like "precedes vowel") could be added pretty easily.

[erogol]

Great news @synesthesiam . Feel free to send a PR ✨

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[skol101]

@synesthesiam sorry for bothering, but am curious if you've pushed the PR?

[synesthesiam]

Hi @skol101, no bother 🙂 Thanks for your patience with this issue.

I did push a PR, but it's since been removed and those changes will be bundled together with fixes/additions for two other issues:

• Whitespace preservation with punctuations
• Support for SSML

You may be able to use my fixes anyway by simply upgrading the version of gruut in your Python virtual environment for 🐸 TTS with: pip3 install --upgrade 'gruut[cs,de,es,fr,it,nl,pt,ru,sv]~=1.3.0'

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.