Improved acoustic model support: introducing autoregressive structure

[r9y9]

As in the shallow AR model proposed by Xin Wang.

The issue was part of #1 but I raised a new issue since this is one of the very important action items to improve the singing voice synthesis quality. Specific discussion and progress can be done in this thread. Welcome any comments and suggestions.

• Shallow AR
• Standard AR
• MDN + AR
• Stream-wise modeling? Stream-wise training #21

[taroushirani]

Hello, I try to change the acoustic model from Conv1dResnet to RMDN as a preliminary step towards Shallow AR and I'm suffered from a shortage of GPU memory.

MDN returns pi, sigma, mu and total number of returned parameters is BxGx(2xD_out + 1) (B is batch size, G is number of Gaussian components, D_out is the number of dimension of target variable). In my experience, the upper limit of this value seems to be about 4000 when using GPU with 8GB memory. Because the out_dim of the default acoustic model is 199, we can set the number of Gaussian components to at most 10 or so, but this may be not enough to get good results.

Is there any need to train f0, mgc, bap separately, or is there good method to reduce memory consumption?

[r9y9]

As I mentioned in #20, the number of Gaussians wouldn't be a large value (>16) in general, so I suppose using a mixture of density networks doesn't increase GPU memory usage dramatically. If we want to save GPU memory usage, we can use a smaller batch size, and it should be okay in my experience. If the batch size matters, we can implement a gradient accumulation trick.

Is there any need to train f0, mgc, bap separately

If we want to try the shallow-AR model, we would need to model F0 separately, at least. I didn't do that for simplicity, and my time was limited. Anyway, It's worth trying. If we implement the separate stream-wise training strategy, we can reduce GPU memory usage accordingly.

[taroushirani]

Thank you for your comment. I'll try the smaller number of Gaussians and investigate gradient accumulation trick.

[r9y9]

I'm revisiting the autoregressive model. I managed to make my implementation work reasonably well, but not significantly better than the current models. #129

Pros

• (Slightly) better temporal dynamics
• (Slightly) less-smoothed outputs

Cons

• Extremely slow training. It takes a week for 100 epochs with Ritsu's database. It depends on settings though.