What to watch and when to stop for fine tuning?

[chigkim]

During the training, it prints out bunch of metrics numbers.
Could you explain in the readme what metric to watch and when to stop?
Thanks!

[leng-yue]

Unfortunately, there is no exact reference number available. My suggestion would be to listen to the generated audio and choose the one that you prefer. Additionally, training for more steps (such as 300k) usually does not negatively impact the quality.

[chigkim]

Thanks for the advice.
Does it generate and save audio file that I can listen during validation? If so, where does it save?
If not, do I need to run inference on a separate machine using as it saves models in logs?
Or, can you run inference on the same machine while it's running fine tuning?

[leng-yue]

It's on wandb / tensorboard.

[chigkim]

Thanks! My Tensorboard wasn't working because I didn't have third party cookies allowed!