Jump in sub_loss/train_dur_loss_step

[vn09]

Hi @p0p4k ,

I hope this message finds you well. I am currently working on training the pflowtts model with my own dataset and have encountered an unexpected behavior that I'm hoping to get some assistance with.

During training, I've observed significant jumps in the sub_loss/train_dur_loss_step metric, as illustrated in the screenshot below:

I have followed the recommended setup and training guidelines, but I am unsure what might be causing these fluctuations. Here are some details about my training configuration and dataset:

   batch_size: 64
   n_spks: 1
   ...
  data_statistics:
    mel_mean: -6.489412784576416
    mel_std: 2.281172275543213

I would greatly appreciate it if you could provide any insights or suggestions that might help resolve this issue. Perhaps there are known factors that could lead to such behavior or additional steps I could take to stabilize the training loss?

[p0p4k]

Could be some anomaly in the dataset. Don't worry about it, let the model train and check inference, that is when you start debugging

[rafaelvalle]

To find problematic samples, one can generate transcriptions with whisper v3 and compare them with the transcriptions in the data by looking for samples with high edit distance for example.

[p0p4k]

@rafaelvalle [unrelated question] given any sliced prompt from the target mel, since they all are supposed to give out the same target mel; is there a way to add some loss for this in one forward pass while using multiple slices for the same target mel? Thanks.

[vuong-ts]

Thanks @rafaelvalle @p0p4k . I use the trained model to run again on training data to filter out outlier samples (dur_loss > 5) and it helps. The loss of new train is smooth now.

[rafaelvalle]

By looking at the text and audio of the samples with dur_loss larger than 5, can you determine the reason for such high loss? Usual suspects include incorrect transcription, long pauses, etc...

…

On Fri, Dec 8, 2023, 08:02 vuong-ts ***@***.***> wrote: Thanks @rafaelvalle <https://github.com/rafaelvalle> @p0p4k <https://github.com/p0p4k> . I use the trained model to run again on training data to filter out outlier samples (dur_loss > 5) and it helps. The loss of new train is smooth now. — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARSFD7TILEEXJFR6AVCGITYIJ35VAVCNFSM6AAAAABAFYZI3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBWGQ3DCMRYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vn09]

The issues mostly are:

• incorrect transcription like wrong text inserted at the beginning
• long pause at the end.