Warning! Reached max decoder steps

[hadaev8]

I had no such problem before
But today used pretrained models
And sometimes spectrogram is bugged

here is notebook for reproduce
https://colab.research.google.com/drive/1jR12cEKdkg0hlDUHGhf2fPb0RwqPwEYj?#scrollTo=CyBu2F7eisFM

[rafaelvalle]

Take a look at the gate outputs and decrease the gate threshold accordingly.

[hadaev8]

@rafaelvalle
But I can't get, why spectrogram different for same text?
And why I never notice it before? Last time used in January.
Maybe it because of fc0d34c ?

Also, gate looks this

[rafaelvalle]

The spectrogram is different with the same text because of the dropout on the Prenet layer: https://github.com/NVIDIA/tacotron2/blob/master/model.py#L100

The gate threshold is what controls when the model should stop decoding. You can decrease the value to prevent such issues.

[hadaev8]

Soo, I get this error with pretrained tacatron and default code from repo, don't you want investigate?

[rafaelvalle]

What sentence did you use?

[hadaev8]

Seems it happens sometimes if I have whitespace at the end, like:
'We have started the invasion. '

[RoelVdP]

I too think this issue may have been closed out too early. I sometimes see this error. Seems to happen with shorter sentences.

I think I found a valid input for which it always happens; try "%1" as the only input. i.e. try and let it speak "%1". Make sure %1 is not interpreted as some variable or something. The sound should be (incorrectly) "one" and then 11-40 seconds of random "singing" noise (and the decoder steps error will show).

This happens with pre-trained model. Reducing gate threshold does not help. Increasing decoder steps sometimes works (but not for this example). It is also starting to look somewhat sporadic/case based.

[RoelVdP]

I tested/confirmed that removing the change in fc0d34c makes no difference; the error still happens.

[shoegazerstella]

I am having the same issue.
My input text is hello, this is the -weird- generated wav file. The good thing is that the tts does not crash, but it seems the last letter o is repeated for 9 seconds.
Would it be possible to avoid this? maybe by attaching some silence at the end of the phrase if this is too short?

[rafaelvalle]

The model is very sensitive to punctuation.
Make sure to trim trailing spaces and always using some punctuation.
In the hello case, try adding the period: "hello."
You can also try modifying the gate threshold.

[haqkiemdaim]

I am having the same issue.
My input text is hello, this is the -weird- generated wav file. The good thing is that the tts does not crash, but it seems the last letter o is repeated for 9 seconds.
Would it be possible to avoid this? maybe by attaching some silence at the end of the phrase if this is too short?

Hi there @shoegazerstella just want to ask, have you solve the decoder issue ? What would be the best solution ?

[rafaelvalle]

@haqkiemdaim
Have you tried adding a '.' to the text input: "Hello." instead of "Hello" ?

[haqkiemdaim]

@haqkiemdaim
Have you tried adding a '.' to the text input: "Hello." instead of "Hello" ?

Haven't yet. But my dataset does not contain any punctuation.

[rafaelvalle]

You can add to every sentence of your dataset a token that signifies EOS (End of Sentence).

[haqkiemdaim]

You can add to every sentence of your dataset a token that signifies EOS (End of Sentence).

ouhh i see. Is it crucial? because in plain tacotron, i used the same data/corpus without punctuation and it went well. But not in Tacotron2. Any reason for that ?

[rafaelvalle]

Tacotron 1 has the EOS token :-)

…

On Sat, Nov 2, 2019, 5:20 PM HaqkiemDaim ***@***.***> wrote: You can add to every sentence of your dataset a token that signifies EOS (End of Sentence). ouhh i see. Is it crucial? because in plain tacotron, i used the same data/corpus without punctuation and it went well. But not in Tacotron2. Any reason for that ? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#174?email_source=notifications&email_token=AARSFDYDX7VQJ5PTZVH6573QRYKK7A5CNFSM4HCLSMAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5H5LQ#issuecomment-549093038>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARSFD4YBD5BETQ6SRGWF6LQRYKK7ANCNFSM4HCLSMAA> .

[rafaelvalle]

You can also try adjusting the gate threshold.

…

On Sat, Nov 2, 2019, 5:20 PM HaqkiemDaim ***@***.***> wrote: You can add to every sentence of your dataset a token that signifies EOS (End of Sentence). ouhh i see. Is it crucial? because in plain tacotron, i used the same data/corpus without punctuation and it went well. But not in Tacotron2. Any reason for that ? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#174?email_source=notifications&email_token=AARSFDYDX7VQJ5PTZVH6573QRYKK7A5CNFSM4HCLSMAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5H5LQ#issuecomment-549093038>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARSFD4YBD5BETQ6SRGWF6LQRYKK7ANCNFSM4HCLSMAA> .

[haqkiemdaim]

@rafaelvalle This is what i have done so far:

• remove trailing space
• change max decoder steps from 500 to 2000

But still reach max decoder steps at 4th epoch.

Now i'm running again my dataset but with punctuation "." at end of sentence for every line.

Will update here the outcomes.

Regarding the gate threshold, currently it is 0.5.

But im a bit confuse, why we need the gate threshold while we already set the max decoder steps to a certain value ?

[SomeUserName1]

@haqkiemdaim Did adding punctuation fix the error?

[shoegazerstella]

@haqkiemdaim Did adding punctuation fix the error?

it worked for my case. thanks!

[thangdc94]

@rafaelvalle
If EOS work why don't you guys make it as default option in symbols.py?
I see other contributors provide answer related to EOS such as accelerate convergence of attention here.

[ErfolgreichCharismatisch]

How do you try except the message "Warning! Reached max decoder steps"? Which file creates it?
EDIT: I found it at https://github.com/NVIDIA/tacotron2/blob/185cd24e046cc1304b4f8e564734d2498c6e2e6f/model.py

[coding-dallas]

I had no such problem before But today used pretrained models And sometimes spectrogram is bugged

here is notebook for reproduce https://colab.research.google.com/drive/1jR12cEKdkg0hlDUHGhf2fPb0RwqPwEYj?#scrollTo=CyBu2F7eisFM

I had no such problem before But today used pretrained models And sometimes spectrogram is bugged

here is notebook for reproduce https://colab.research.google.com/drive/1jR12cEKdkg0hlDUHGhf2fPb0RwqPwEYj?#scrollTo=CyBu2F7eisFM

the notebook link is not opening