[Feature] Support EarlyStoppingHook

[nijkah]

Motivation

Closes #356

Modification

Added a new variable stop_training in EpochBasedTrainLoop and IterBasedTrainLoop.
Edited run method as

    def run(self) -> torch.nn.Module:
        """Launch training."""
        while self._epoch < self._max_epochs and not self.stop_training:
            self.run_epoch()

**
EarlyStoppingHook itself does not save the best checkpoint. Should we support it?

I referred to some logics from

• https://github.com/Bjarten/early-stopping-pytorch/blob/master/pytorchtools.py
• https://docs.ray.io/en/latest/_modules/ray/tune/stopper/experiment_plateau.html#ExperimentPlateauStopper

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (main@25dfe41). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head fe6ff30 differs from pull request most recent head 046d7be. Consider uploading reports for the commit 046d7be to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #739   +/-   ##
=======================================
  Coverage        ?   76.71%           
=======================================
  Files           ?      139           
  Lines           ?    10921           
  Branches        ?     2184           
=======================================
  Hits            ?     8378           
  Misses          ?     2186           
  Partials        ?      357           

Flag	Coverage Δ
unittests	76.71% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[nijkah]

Hi @HAOCHENYE, I have a little question.
In the UT with initializing Runner, the error raises when the experiment_name of Runner isn't specified. Also, it happens when the experiment_name is duplicated with the other UTs although there is a logic for cleaning its temporary directory.

Is it intended (the developers should consider) or unintended behavior?

[HAOCHENYE]

Hi @HAOCHENYE, I have a little question. In the UT with initializing Runner, the error raises when the experiment_name of Runner isn't specified. Also, it happens when the experiment_name is duplicated with the other UTs although there is a logic for cleaning its temporary directory.

Is it intended (the developers should consider) or unintended behavior?

Usually, if the user does not specify the experiment name, Runner will automatically generate the experiment name and each Runner instance will have unique experiment names with different timestamp suffix. However, if the time interval between Runner's creation is so short that the timestamp suffixes are identical, it could lead to Runners with duplicate experiment names, which is not allowed in current design.

MMEngine has multiple global variances such as MessageHub, Visualizer and DefaultScope, and all these variables are marked with the experiment name of Runner. Global variables are very dangerous, therefore these variables must have unique names.

As for the unit tests will often raise an error about the duplicated experiment name, it is caused by the irregularity of the current unit testing, and we plan to refactor the unit tests, and provide a more general function to build Runner in unit tests.

[nijkah]

One more question! I am thinking about which experiments we should consider for no improvement.

Current implementation retains top scores ignoring the recent inferior results.
But it may be more intuitive to retain latest scores to check the trend of training.

Which one will be better?

[HAOCHENYE]

One more question! I am thinking about which experiments we should consider for no improvement.

Current implementation retains top scores ignoring the recent inferior results. But it may be more intuitive to retain latest scores to check the trend of training.

Which one will be better?

Additional notes： We could also borrow the implementation in

https://github.com/Lightning-AI/lightning/blob/9a4e8a8c528d6475ab33c46a0a84e27273cc10bd/src/pytorch_lightning/callbacks/early_stopping.py#L38

Back to the topic, it could make more sense to me to directly compare the latest metric with the previous best Nth metrics and finally update the Nth metrics with the latest one, rather than first update the Nth metrics, and then compare them 🤣. what do you think about it?

[nijkah]

I think your provided link looks neat. I'll rewrite the code especially for stopping logic and __init__ parameters.

[zhouzaida]

Hi @nijkah , thanks for your contribution. We are planning to merge this PR today and release a new version 0.6.0. The feature to resume can be supported in the future if users require it. In addition, this PR can be merged after resolving the conflict.

@zhouzaida I apologize for not finishing the rest work. I think it is okay to merge right now. I'll try to stay tuned to the future work for this.

Hi @nijkah , thanks for your contribution again 👍.