MDN implementation

[taroushirani]

Hello, I made sample implementation of Mixture Density Networks for better timelag prediction following Sinsy[1].
Contrary to my expectations, there is no statistically significance among the current timelag model and MDN-based models.
(As for duration model, the results of MDN-based models are poorer than that of conventional one(see statistical analysis below)).

I checked my code repeatedly but I could not find my mistakes. Could anyone please advise me to improve the quality?

1. http://www.apsipa.org/proceedings/2018/pdfs/0001003.pdf

In this experiment I use NIT-SONG070 data-set, and the number of Gaussian components of MDN and RMDN model are
set to 500, 100 respectively.

Statistical analysis

• Normality test
  timelag: ShapiroResult(statistic=0.943882942199707, pvalue=9.166962328110984e-12)
  duration: ShapiroResult(statistic=0.6752718687057495, pvalue=1.4900654531575684e-37)

Errors seems not to be distributed normally, so we use a non-parametric test.

• Kruskal-Wallis test
  timelag:KruskalResult(statistic=0.12859137306783802, pvalue=0.9377276850847137)
  duration:KruskalResult(statistic=18.355098064661238, pvalue=0.00010333349007178473)

There seems to be statistically significance among duration errors of these models.

• Conover’s test

duration	Conventional	MDN	RMDN
Conventional	1.000000	0.000115	0.000333
MDN	0.000115	1.000000	0.785517
RMDN	0.000333	0.785517	1.000000

The conventional model is statistically more accurate than other 2 models.

[r9y9]

Thanks for your great contribution! I added a few comments on numerical stability and efficiency. Besides, I have one suggestion to make it work better: use a small number of Gaussians! 500 sounds extremely large to me. In my experience, 10 would be good for a general choice.

I looked into the Hono's paper again and they used 1 mixture component. This is reasonable to me for duration modeling since the output is 1 dimension and the task is not very complicated. We might want to increase the number of Gaussians for acoustic modeling though.

For reference, https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/42020.pdf, Heiga uses 1-to-16 mixtures for DNN-based TTS task. According to the paper, 2 mixture was good enough (8 mixture was the best).

So, could you try adjusting the number of Gaussians? 1, 2, 4, 8, 16 would be good to go. You might feel 1 is not very effective, but sometimes it works comparable to 10 mixtures and training becomes very stable. https://arxiv.org/abs/1807.07281

[r9y9]

Looks like this is numerically unstable. Could you consider using logsumexp (https://pytorch.org/docs/stable/generated/torch.logsumexp.html)? You can refer my implementation https://github.com/r9y9/wavenet_vocoder/blob/63d3fe0a85a69aa25d80700974fe282c96fb6b65/wavenet_vocoder/mixture.py#L202-L218 as an example.

Ref: https://blog.feedly.com/tricks-of-the-trade-logsumexp/

[taroushirani]

Thank you for your comment. I changed mdn_loss according to your advice.

[r9y9]

Looks like this is not efficient... Can we do better at this?

https://github.com/r9y9/wavenet_vocoder/blob/63d3fe0a85a69aa25d80700974fe282c96fb6b65/wavenet_vocoder/mixture.py#L221-L270 I implemented sampling from Gaussian mixtures a while ago. It might be useful as a reference.

[taroushirani]

Thank you for your comment. Do you mean I should do this without for loop, or I should do this in more probabilistic manner? I borrow to_one_hot from your wavenet_vocoder/wavenet_vocoer/mixture.py and re-implement this without for loop. And I add mdn_sample to sample predicted variables in probabilistic manner.

[r9y9]

I meant it would be great removing the for loop for efficiency. Taking a mean is okay.

[taroushirani]

Thank you for your comment. I changed the code according to your advice and add some test code (and sorry for forgetting squash merge). I'll try smaller number of Gaussians later.

[taroushirani]

I did the same experiment once again with smaller number of Gaussian components. I made use of NIT-SONG070 data-set and each model has 2 hidden layers with 128 dims. There seems no statistical significance in timelag model, but some models in duration are statistically poorer than conventional one.

Statistical analysis

Normality test

timelag: ShapiroResult(statistic=0.922505795955658, pvalue=6.90394944215795e-28)
duration: ShapiroResult(statistic=0.8082517385482788, pvalue=0.0)

Errors seems not to be distributed normally, so I use a non-parametric test.

Kruskal-Wallis test

timelag:KruskalResult(statistic=12.580251469105907, pvalue=0.24809602556265878)
duration:KruskalResult(statistic=133.5905905860436, pvalue=8.636478810387355e-24)

There seems to be statistically significance among duration errors of these models. According to the results of Conover's test, mdn_mix4, rmdn_mix4, rmdn_mix16 are not lesser than conventional one.

I also recorded the transitions of loss of these models.

In timelag model, the learning process seems not to work well in MDN model, and over-fitting occurs in RMDN model.

[taroushirani]

I fixed the bug of nnsvs/bin/synthesis.py which fails to synthesis when label_path and out_wav_path are specified instead of utt_list etc. And I don't mind waiting. I'm in no hurry.

[taroushirani]

I have some questions about the implementation of mixture density network.

(1) I changed pi of MDNLayer into the not softmax-ed version for numerical stability and currently it has no mathematical meaning in itself(unless it is softmax-ed by hand). Is it better to change pi into logpi(logged "pi") by adding F.log_softmax to MDNLayer.forward() ?

(2) I found that mdn_loss sometimes goes to nan when num_gaussians is large. In my knowledge, MDN won't work if one of its variance goes to zero and its Gaussian component shrink to a point. Your implementation[1] has the lower limit of log_scale(logged variance) and I guess this is to avoid the collapse of MDN. Is this correct?

(3) In your implementation you use logged version of scale. Is this for numerical stability?

1. https://github.com/r9y9/wavenet_vocoder/blob/63d3fe0a85a69aa25d80700974fe282c96fb6b65/wavenet_vocoder/mixture.py#L161

[r9y9]

(1) Yes, I think we should apply F.log_softmax to have mixture weights to be consistent with the mathematical formulation.

(3) Yes. Otherwise, the scale would quickly go to zero, especially when the number of Gaussians is large.

(2): The log scale clipping is for stabilizing the training. I recommend using both 1) log-scale parameterization and 2) clipping to improve numerical stability.

[taroushirani]

Thank you for your rapid comment. I changed to use log_pi and log_sigma instead of pi and sigma as you recommended.

And I have another question. I'm afraid that torch.distributions.Normal.log_prob(x) may return -inf when the possibility of x is zero. Is there any need to clamp the result of log_prob with the same way?

[taroushirani]

torch.distributions.Normal.log_prob(x) may return -inf even if the possibility is very small but not zero, due to the precision of torch.tensor(pytorch/pytorch#12269)

[r9y9]

I think it's a common issue. Adding a clamp should fix the numerical stability.

[taroushirani]

Thank you for your comment. I tried to clamp the result of torch.distributions.Normal.log_prob(x) but the numerical instability worsened. I have no idea why this occurred and my codes seems to be numerically unstable yet.

[r9y9]

Umm. I will also look into the issue this weekend!

[taroushirani]

I added torch.autograd.set_detect_anomaly(True) to nnsvs/bin/train.py and I found that the back propagation of torch.logsumpexp(=LogsumexpBackward) returned nan.

In my knowledge, weight coefficient pi must be between 0 and 1 due to the nature of the mixture density network, and this can apply to torch.distributions.Normal.

-inf < log_pi < 0
-inf < torch.distributions.Normal.log_prob(y) < 0

So the summation of them must be between -inf and 0.

If x can be written as x = (x_1, x_2,...,x_n) the partial derivative of logsumexp(x) (d logsumexp(x) /dx_i) is 1/(sum exp(x)) * exp(x_i).
So if x = -inf, the back propagation of logsumexp(x) means division of 0 by 0 and will return nan.

This may be solved by setting lower limits to all logged values. (I thought F.log_softmax would be implemented with consideration of such a situation like this, but merely clamping log_pi solved my problem...)

[r9y9]

Hi, I am going to look into this very soon, hopefully tonight or tomorrow. I will first fix #14 and refactor recipes to reduce dup across recipes. Then, I will review the PR and suggest some changes to make it into good shape. Thanks for your patience!

[r9y9]

First of all, thank you for your great effort! Overall it looks very good. I tried the PR locally and it worked also well. Please see the individual comments to make the code better.

For numerical instability, I cannot reproduce the issues locally. I tried to uncomment some of the clamp operations, but it just worked okay. If you can provide me detailed steps to reproduce, I can look into it more carefully.

For the commit history, it would be very helpful if you cleaned it up (maybe using git rebase?), so that I can review it easily. That said, I understand using git is a bit difficult (even for me!). I am okay if you happen to mess up the commit history. In that case, I can run git rebase when I merge the PR if necessary.

Many thanks!! I am looking forward to getting the MDN feature into the master.

[r9y9]

Please consider following #25 and #26 to avoid a dup.

[taroushirani]

I deleted the duplication in the latest commit.

[r9y9]

With #26, these configs can be implemented by an external config. See https://github.com/r9y9/nnsvs/tree/caefa1081bbd8d62ab9eb945033bd3ea02814d18/egs/nit-song070/00-svs-world/conf/train/model for example.

[taroushirani]

I deleted the duplication in the latest commit.

[r9y9]

Same as above. Also, tabs and whitespaces are mixed. Please use whitespace for consistency.

[taroushirani]

I deleted the duplication in the latest commit.

[r9y9]

Same as above.

[taroushirani]

I deleted the duplication in the latest commit.

[r9y9]

This file is a dup. Please use a symlink instead, or extend the existing 00-svs-world recipe with external config files.

[taroushirani]

I changed to use symlink in the latest commit.

[r9y9]

I think the first line is unnecessary. Please consider replacing the second line with from nnsvs import mdn

[taroushirani]

I changed import directive as you recommended in the latest commit.

[r9y9]

Please consider changing this like:

if not model.is_deterministic() or if model.prediction_type() == ENUM_VALUE

[taroushirani]

I changed to use predictin_type() of BaseModel in the latest commit.

[r9y9]

Same as above.

[taroushirani]

I changed to use predictin_type() of BaseModel in the latest commit.

[r9y9]

Same as above.

[taroushirani]

I changed to use predictin_type() of BaseModel in the latest commit.

[r9y9]

Same as above.

[taroushirani]

I changed to use predictin_type() of BaseModel in the latest commit.

[r9y9]

The code is getting in a good shape! I left a few minor comments. Once the minor issues are addressed, I am going to merge the PR.

[r9y9]

Looks like this is not used anymore?

[taroushirani]

I deleted unnecessary ReLU in the latest commit.

[r9y9]

minor: could you keep the new line here?

[taroushirani]

I added new line there in the latest commit.

[r9y9]

Same as above

[taroushirani]

I added new line there in the latest commit.

[r9y9]

Same as above

[taroushirani]

I added new line there in the latest commit.

[r9y9]

Same as above

[taroushirani]

I added new line there in the latest commit.

[r9y9]

LGTM. Let's merge and play with it.

[r9y9]

Many thanks for your great contribution!!

[taroushirani]

Thank you for your code review. I probably fixed all problems you pointed out above.

I met numerical instability when I tried Conv1dResnetMDN(namely Conv1dResnet + MDN). I'll submit PR after MDN is merged.

As far as I know, git rebase re-assign new hash to each commit and make the log straight, but it will destroy the repository which refers to the old hashes. To the amateur eye, I feel "squash merge" is safer than rebase.