Can't find Max training steps field

[am576]

I'm using iterational approach in my trainings using small number of Training Steps Per Image with each iteration, starting another one if I am not pleased with the outcome. E.g starting with 80 s/i, after training is complete I can run another 20 etc.
But somehow now I can not start training. I got model with 1800 steps, I'm trying to start another 20 steps iteration, but it won't happen - It starts and stops, displaying a message "Training finished. Total lifetime steps: 1800"
So, is there a way to set max steps to -1 or really high number to allow me to run the iterations as long as I need?

[minienglish1]

Epoch is the equivalent of training steps for the GUI: images * epochs = training steps, at least for batch size 1 anyway.

When you hit train, it trains for how many epochs you selected, regardless of what it has trained in the past. Or that's been my experience at least.

I train for 40,000 steps at minimum when I train, so I don't understand how how you're having trouble going past 1800. I usually just put in 1000 epochs, save checkpoint every X epochs, then go to sleep/work and compare the checkpoints later. If I need it to train more, I just hit train again.

I don't think the gui has training steps selectable anymore. But I could be wrong. Maybe try updating the extension?

[am576]

Well, usually I have no troubles going as far as I need. But once in a while the menthioned issue appears when I cannot go past the initial number of calculated max steps. I'm not sure how to reliably reproduce it so I can post an issue

[Arilziem]

As @minienglish1 has pointed out, previous training steps are not taken into account, they are just for informational purposes.

This could be a VRAM issue. When this extension encounters OOM, it will try to reduce the batch size. With a batch size of 1, it used to reduce it to 0, and then concluded there was nothing to be done (and so training was "finished"). I am pretty sure this was changed a few weeks ago (because that was pretty confusing behavior), but I am not sure if this has landed in main branch already.

If you encounter this again, post the log and your training settings here, and what type of GPU you are using.

[am576]

Thanks for the suggestion, but it's doubtful to be a hadrware related issue. I'm using a T4 gpu with 32gb of system ram, and have hever encountered any OOM or Cuda crashes.
Sure, I'll copy the log next time(hopefully not) this happens

[minienglish1]

Check the json file (db_config.json) of the model your training to see if it has anything about "max_steps." It is located in the stable-diffusion-webui/models/dreambooth/"model_name" directory.

Mine doesn't. If yours does then you're probably on an old version of the extension.

[am576]

I've checked db_json in all the models I'm currently training. None of them has max_steps or similar key.
Well, I assume it was a glitch, but nonteless will keep watching

[minienglish1]

good luck

[Arilziem]

Thanks for the suggestion, but it's doubtful to be a hadrware related issue. I'm using a T4 gpu with 32gb of system ram, and have hever encountered any OOM or Cuda crashes

The T4 has 16GB of VRAM, no? The memory management has been somewhat non-ideal on main branch for around 2-3 months, so depending on your training settings and what else is using VRAM, OOM would absolutely be a possibility. It would also explain why it only happens sometimes. It should still output a CUDA error though. In any case, logs and so on would help to see what's going on.