about 'An Easier Trick for Boundary Prediction' #17
Comments
Hi, I'm celebrating the Chinese Spring Festival these days. 🤗 |
|
Thanks for the quick reply even on the celebrating day! Yes I understood what you mean, but actually I double-checked that the predictor consumes teacher-forced mel which is generated by inputting gt F0. duration_target is not None: True
f0 is not None: TrueI used this function in def q_mean_variance(self, x_start, t):
mean = extract(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start
variance = extract(1. - self.alphas_cumprod, t, x_start.shape)
log_variance = extract(self.log_one_minus_alphas_cumprod, t, x_start.shape)
return mean, variance, log_varianceso that the expected KLD of step @torch.no_grad()
def expected_kld_T(self, x_start, mask, noise=None):
t = self.num_timesteps # t = T
x_start, t, mask = self.kld_input(x_start, t, mask)
mu, _, logvar = self.q_mean_variance(x_start, t)
mu, logvar = (mu.squeeze(1) * mask), (logvar.squeeze(1) * mask)
kld = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
kld = kld / mask.sum()
return kldbut the results show more than 10 times smaller than every expected KLD of step @torch.no_grad()
def expected_kld_t(self, x_pred, x_gt, t, mask):
x_pred, t, mask = self.kld_input(x_pred, t, mask)
x_gt, *_ = self.kld_input(x_gt)
coef = extract(self.alphas_cumprod / (2 * self.log_one_minus_alphas_cumprod.exp()), t, x_pred.shape)
kld = F.mse_loss(self.noised_mel(x_pred, t), self.noised_mel(x_gt, t), reduction='none')
kld = (kld * mask).sum() / mask.sum() # or kld.mean() ?
kld = coef[0].squeeze() * kld
return kldwould it be matter for the issue? |
Have you made sure that ~M is correct? We calculated this k a few months ago, and we may have made some approximations at that time. If your problem hasn't been solved, I'll check it again. |
|
which part do i have to check in addition to the ground-truth F0 for ~M? @torch.no_grad()
def kld_input(self, x, t=None, mask=None):
x = self.norm_spec(x)
x = x.transpose(1, 2)[:, None, :, :] # [B, 1, M, T]
if t is not None:
t = torch.ones(x.shape[0], device=x.device).long() * (t-1)
if mask is not None:
mask = ~mask.unsqueeze(-1).transpose(1, 2)
return x, t, maskdef norm_spec(self, x):
return (x - self.spec_min) / (self.spec_max - self.spec_min) * 2 - 1 |
Hello, I am also following this problem, have you solved it? |
In your paper, we can get the predicted boundary as follows:
then I implemented 'An Easier Trick for Boundary Prediction' in my repo following the trick:
https://github.com/keonlee9420/DiffSinger/blob/f849f8def5abb38ad272a384e8bec838ea1957a4/boundary_predictor.py#L14-L45
and there are some helper functions for that (please focus on
expected_kld_tandexpected_kld_Tfunction):https://github.com/keonlee9420/DiffSinger/blob/f849f8def5abb38ad272a384e8bec838ea1957a4/model/diffusion.py#L351-L389
But as I noted in my README.md (in
2.of note section), the predicted boundary of LJSpeech is 100, which is the same as the total timesteps in Naive version.So I'd like to ask you to briefly check my implementation. Could you please take a look at it and let me know if I missed something? Why do you think my boundary predictor shows unexpected
K_step?FYI, here is the sample output log of running
boundary_predictor.py:Thanks in advance!
The text was updated successfully, but these errors were encountered: