Allow disabling automatic stopping during fitting

[EricWiener]

🚀 Feature

Currently if neither max_epochs nor max_steps aren't set, lightning defaults to using max_epochs = 1000. However, in some situations the user actually doesn't want max_epochs or max_steps to be set, and automatically stopping after 1000 epochs isn't wanted. It would be great if there were a way to specify that no epochs should be set.

Motivation

I was running a very large training job over the weekend with a very large dataset. Because the dataset is so large, I set the number of batches per epoch to be very small (relative to the size of the dataset) so that logging would occur more frequently. I set max_epochs and max_steps to be None because I believed this would disable automatic stopping, on Monday when I checked on the model again it had exited early after only a couple of hours.

Pitch

Have some way to specify that automatic stopping should be disabled. Having to hard-code a large number like 2**1000 isn't the most elegant (especially compared to how elegant the rest of Lightning is).

Alternatives

The user could pass float("inf") as max_epochs to disable stopping if not using multiple GPUs. However, if using multiple GPUs they would need to use a very large integer instead (ex. 2**1000).

If using a float for max_epochs, you will get the following error:

Training: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/home/eric/.../train.runfiles/train.py", line 371, in <module>
    main(sys.argv[1:])
  File "/home/eric/.../train.runfiles/train.py", line 362, in main
    trainer.fit(model, data_module)
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/home/eric/.../train.runfiles/pypi__pytorch_lightning_python3_deps/pytorch_lightning/trainer/trainer.py", line 861, in run_train
    epochs = range(self.current_epoch, self.max_epochs) if self.max_epochs else count(self.current_epoch)
TypeError: 'float' object cannot be interpreted as an integer

Additional context

https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1627425271133200

[ananthsub]

@awaelchli this is similar to #8210 (comment)

[EricWiener]

Just as a follow-up, I still think the default should be jobs are automatically stopped to avoid forgotten jobs racking up lots of server costs. However, I think the change in behavior should specifically clean up how to specify you actually want the job to run forever.

[awaelchli]

I think Lightning's 1000 epochs is a safety feature.
The only idea I have without changing this behavior is to add new special default:

class Trainer:
def __init__(..., max_epochs: Optional[Union[X, int]] = X): 

Where X is a sentinel value (is this the right term??) to differentiate from None. And when user sets it to None we have the infinite loop, but if not we keep the previous behavior. Not the most elegant but that's what I can think of now

[EricWiener]

@awaelchli do you think maybe setting max_epochs to a negative value?

[awaelchli]

that could also work yes ^^

[EricWiener]

Awesome. I'll create a PR

[EricWiener]

@awaelchli any recommendations for how to check the trainer will train forever if max_epochs is negative?

Also, I was debating whether to allow both max_epochs and max_steps to be set to negative/have the other be None to disable automatic stopping. This would mean the following combos would disable automatic stopping: (negative, negative), (negative, None), (None, negative), but this seemed like it would just be a mess to maintain and wasn't worth it, so I left it with just having max_epochs be set to negative + max_steps needs to be None. Thoughts?

[awaelchli]

@EricWiener actually I believe we should also introduce max_steps=-1. Then:

max_epochs, max_steps, effect
-1, -1, trainer will not stop unless external forces
-1, Any, turns off limits for epochs, still limited by max_steps
Any, -1, the behavior we know today

[EricWiener]

Current behavior is:

# trainer.py
fit_loop = FitLoop(
    min_epochs=(1 if (min_epochs is None and min_steps is None) else min_epochs),
    max_epochs=(1000 if (max_epochs is None and max_steps is None) else max_epochs),
)

#fit_loop.py
stop_steps = self.max_steps is not None and self.global_step >= self.max_steps
stop_epochs = self.max_epochs is not None and self.current_epoch >= self.max_epochs

So if we were to allow (max_epochs = Any, max_steps = -1), we would immediately stop training. Same would be true if we allow (max_epochs = -1, max_steps = Any). The logic would need to be changed in order to allow both max_epochs and max_steps to be -1 (and not immediately stop training). I was trying to avoid adding additional complexity. What do you think?

[ananthsub]

@EricWiener @awaelchli are there other validations we then need to add for min_steps and min_epochs?
For example, I think we should fail early if someone specifies (min_epochs=number greater than 0, max_epochs=-1)

[EricWiener]

@EricWiener @awaelchli are there other validations we then need to add for min_steps and min_epochs?
For example, I think we should fail early if someone specifies (min_epochs=number greater than 0, max_epochs=-1)

What would be the issue with allowing min_epochs = number greater than 0 and having max_epochs = -1? Setting max_epochs = -1 would disable automatic stopping based on max_epochs/max_steps, but it wouldn't disable stopping based on EarlyStopping or max_time. It could still make sense to set a desired minimum number of epochs to be met.

[ananthsub]

What would be the issue with allowing min_epochs = number greater than 0 and having max_epochs = -1? Setting max_epochs = -1 would disable automatic stopping based on max_epochs/max_steps, but it wouldn't disable stopping based on EarlyStopping or max_time. It could still make sense to set a desired minimum number of epochs to be met.

Yes you're absolutely right, my mistake.

As there are multiple stopping conditions one could set with either max_epochs, max_steps or max_time - which one takes precedence? What if a user specifies max_epochs=-1 and max_steps = number > 0 ? or vice versa? or if max_epochs=-1 and max_time=timedelta(...) ?

I would prefer to raise a misconfiguration exception in case we have conflicting bounds instead of trying to determine some precedence across these conditions.

FWIW, another opportunity or infinite training is online training, where new data is continuously streamed in and we don't want to stop even if the dataloader is tempoarily exhausted