Continue Jeremy's early stopping PR #1504

[awaelchli]

What does this PR do?

This is just a continuation of #1504 as now it is their repo branch and much easier for @PyTorchLightning/core-contributors cot access :]

Fixes #1464
Fixes #1463
Fixes #1699
Fixes #2151
Related #1458

Just trying to help complete #1504

TODO:

• test_resume_early_stopping_from_checkpoint
• test_wandb_pickle

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[jeremyjordan]

thanks for carrying this over the line @awaelchli :)

[Borda]

thanks for carrying this over the line @awaelchli :)

add @jeremyjordan that is the Lightning spirit I love... 💓

[awaelchli]

@williamFalcon as requested I added a test to make sure checkpointing only happens once in ddp (ddp_cpu) mode (i.e. they don't get overwritten by other processes).

regarding loggers, on master (and here) when launching ddp with e.g. 2 processes the logger will still create to folders version_0 and version_1. I'd rather not try to fix it here and make the PR even bigger. I can open an issue for this so we can track it down, ok?

[mergify]

This pull request is now in conflict... :(

[mergify]

This pull request is now in conflict... :(

[williamFalcon]

great work everyone!!!

[lucmos]

Hi guys, sorry if I chime in.

I noticed that in lightning 0.8.4 the early stopping saves the last model, not the best one.
"Saves" meaning that when the fit function early stops, the model weights are not the best ones but the ones in the last epoch.

Is this solved in this PR or should I open a new issue?