Getting output/Stopping training mid-process

[Kacperskyy]

Hi, I am currently training for 3125 epochs and eval_interval set to 12500 steps (half of the total)

I got the model from step 12500 and its almost good, but I would like it to be trained a little longer. But I also don't want to sit through the rest of the training.

So I am wondering if there is a way to somehow force the training to stop and output the model as it is instead of waiting to reach the eval_interval steps. (And perhaps continue the training later?)

[Lordmau5]

Yes. Simply lower eval_interval.

That's what's in charge of the checkpoints.

I usually set mine to 100 so I can see faster updates when using the Tensorboard (since it has an audio tab and is doing validation / inference on 2 files there)

epochs is what's in charge of the maximum training length / epochs.

Even if it reaches the max epoch you can just go into the config (main folder -> configs\44k\config.json) and increase it so you can train it further

[Kacperskyy]

Can I just change it during this training session and it will apply instantly?

Because I want to end (and output the model) from the training that I currently run on my PC

[Lordmau5]

No, from what I know it only loads those settings before it starts training, so you'd have to wait for it to finish.

[Kacperskyy]

Damn, that's what I was afraid of. Well thanks for the help anyway!