About question of code and synthesis

[Dyongh613]

HI@keonlee9420, Thank you for your suggestions these days, I successfully integrated model PortaSpeech on the basis of this model. These are some questions to ask you! Thank you!

1. In the DiffGAN-TTS, the return of get_mask from length is mask. And the return of get_mask from length in PortaSpeech is ~mask. I want to know the difference between them,
2. In DiffGAN-TTS, about def diffuse_trace(self, x_start, mask). I want to know how do the ~ aims to do in def diffuse_trace. In my integrated model, I set the return of get_mask from length is ~mask.
   If I delete the ~ in diffuse_trace, the synthesis mel is error and the voice likes to the voice of water. While If I preserve the ~ in diffuse_trace, the mel is also error and the voice likes to electric voice.
   Thank you very much!

• Deng Yan
• 2022.5.9
• GuangXi University

[keonlee9420]

Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part?
Besides, here is my answer to your questions:

1. ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True]
2. so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme.

[Dyongh613]

***@***.***, most of the return of diffuse_trace is mostly zeros or ones. Like this.

…

------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dyongh613]

***@***.***, I see the mel_predictor in aux model is the return of diffuse_trace. After trained with 1000 steps, most of the trace is (-1.5,1.5). The synthesis spectrogram likes 

…

------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dyongh613]

***@***.***, I use the PortaSpeech replacing the FastSpeech2 in DiffGAN-TTS. I find after the decoder, most of the output tensor of FastSpeech2 are (-1,1), while the PortaSpeech output tensor is (-11,11). I see the synthesis voice of problem due to the input(coarse_mel) of diffuse_trace. I cannot deal with it my dear friend.Thank you very much!

…

------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dyongh613]

***@***.***, my friend. I got the voice in shallow module in my model. Best wishes to you my friend! 

…

------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[keonlee9420]

Closed due to inactivity.

[Dyongh613]

Hi @keonlee9420, I just replace FastSpeech2 with PortaSpeech in the Acoustic Generator adjusting the loss and cwt including the energy and pitch. The diffusion part is the same with DiffGAN-TTS. The answer for you said that I have understand. Thank you very much. In PortaSpeech, if I delete the ~ in get mask from length, the loss will become nan in the model. I see I need to adjust the mask set. I see the biggest problem is the difference in the return of get mask from length between DiffGAN-TTS and PortaSpeech.  By the way, I want to know the meaning of the diffuse_trace and diffuse_fn. I'm trying to deal with these problems now. 

…

------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dyongh613]

***@***.***, I met some questions.  1. This is aux model mel trained with 5000 steps. The voice is all electric current. 2.In tensorboard, the sampled spectrogram is the same as GT.  3.The voice made by shallow model likes to  water or electric. ------------------ 原始邮件 ------------------ 发件人: "keonlee9420/DiffGAN-TTS" ***@***.***>; 发送时间: 2022年5月10日(星期二) 晚上11:19 ***@***.***>; 抄送: "Rui ***@***.******@***.***>; 主题: Re: [keonlee9420/DiffGAN-TTS] About question of code and synthesis (Issue #6) Hi @qw1260497397 , thanks for your attention. It sounds interesting integrating PortaSpeech into DiffGAN-TTS, but don't have idea on it. Could you please elaborate it more? So you add GAN training for PortaSpeech? What about diffusion part? Besides, here is my answer to your questions: ~ means 'not' in boolean so if you add it in front of mask, then the mask will be toggled. for example, if a mask value is [True, True, False], then the value of '~mask' is [False, False, True] so the meaning of ~ in diffuse_trace is the same. the error that you report may be raised from the mismatch of masking value usage between PortaSpeech and DiffGAN-TTS, where you have to sync to have the same masking scheme. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>