Find last checkpoints on restart #14907

otaj · 2022-09-27T13:37:38Z

What does this PR do?

Although #12816 enabled support for passing "last" as an argument for reloading checkpoint, reloading happened only for FT checkpoints and checkpoints that were created within the same run.

This PR adds support for reloading last checkpoint on restart of the run.

Note, that this makes "best" and "last" keywords divergent in their behavior - "best" doesn't reload the best available checkpoint when starting a new run for training.

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda @awaelchli @ananthsub @ninginthecloud @rohitgr7 @otaj @carmocca @jjenniferdai @akihironitta

src/pytorch_lightning/callbacks/model_checkpoint.py

src/pytorch_lightning/trainer/connectors/checkpoint_connector.py

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

carmocca

LGTM. Waiting on a test though!

awaelchli · 2022-09-27T20:27:04Z

I found some tests related to this in test_trainer.py, test_checkpoint_path_input_last etc.. Perhaps one could modify one of these or follow a similar pattern as these tests.

otaj · 2022-09-30T12:37:36Z

@awaelchli, @carmocca there is a test now. Btw, writing it helped me uncover a small bug with the code, see d3a3b35

tests/tests_pytorch/trainer/test_trainer.py

awaelchli

Great!

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

* use more recent lightning cloud launcher * allow LightningApp to use custom cloud compute for flows * feedback from adrian * adjust other cloud tests * update * update * update commens * Update src/lightning_app/core/app.py Co-authored-by: Sherin Thomas <sherin@grid.ai> * Close profiler when `StopIteration` is raised (#14945) * Find last checkpoints on restart (#14907) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> * Remove unused gcsfs dependency (#14962) * Update hpu mixed precision link (#14974) Signed-off-by: Jerome <janand@habana.ai> * Bump version of fsspec (#14975) fsspec verbump * Fix TPU test CI (#14926) * Fix TPU test CI * +x first * Lite first to uncovert errors faster * Fixes * One more * Simplify XLALauncher wrapping to avoid pickle error * debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug commit successful. Trying local definitions * Require tpu for mock test * ValueError: The number of devices must be either 1 or 8, got 4 instead * Fix mock test * Simplify call, rely on defaults * Skip OSError for now. Maybe upgrading will help * Simplify launch tests, move some to lite * Stricter typing * RuntimeError: Accessing the XLA device before processes have spawned is not allowed. * Revert "RuntimeError: Accessing the XLA device before processes have spawned is not allowed." This reverts commit f65107e. * Alternative boring solution to the reverted commit * Fix failing test on CUDA machine * Workarounds * Try latest mkl * Revert "Try latest mkl" This reverts commit d06813a. * Wrong exception * xfail * Mypy * Comment change * Spawn launch refactor * Accept that we cannot lazy init now * Fix mypy and launch test failures * The base dockerfile already includes mkl-2022.1.0 - what if we use it? * try a different mkl version * Revert mkl version changes Co-authored-by: awaelchli <aedu.waelchli@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> * Trainer: fix support for non-distributed PyTorch (#14971) * Trainer: fix non-distributed use * Update CHANGELOG * fixes typing errors in rich_progress.py (#14963) * revert default cloud compute rename * allow LightningApp to use custom cloud compute for flows * feedback from adrian * update * resolve merge with master conflict * remove preemptible * update CHANGELOG * add basic flow cloud compute documentation * fix docs build * add missing symlink * try to fix sphinx * another attempt for docs * fix new test Signed-off-by: Jerome <janand@habana.ai> Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: Sherin Thomas <sherin@grid.ai> Co-authored-by: Ziyad Sheebaelhamd <47150407+ziyadsheeba@users.noreply.github.com> Co-authored-by: otaj <6065855+otaj@users.noreply.github.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Jerome Anand <88475913+jerome-habana@users.noreply.github.com> Co-authored-by: awaelchli <aedu.waelchli@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: DP <10988155+donlapark@users.noreply.github.com>

Find last checkpoints on restart

a24e309

otaj added bug Something isn't working checkpointing Related to checkpointing callback: model checkpoint pl Generic label for PyTorch Lightning package labels Sep 27, 2022

otaj added this to the pl:1.7.x milestone Sep 27, 2022

otaj self-assigned this Sep 27, 2022

otaj requested review from tchaton, carmocca, awaelchli, williamFalcon and kaushikb11 as code owners September 27, 2022 13:37

changelog

4909803

otaj requested review from Borda, justusschock and rohitgr7 as code owners September 27, 2022 13:39

otaj added 4 commits September 27, 2022 16:33

check if dir exists

d9e0a70

add broadcast flag to resolve dir

e57f7ae

fix path on windows

25e0067

fix path on windows second attempt

2ae4b64

awaelchli reviewed Sep 27, 2022

View reviewed changes

src/pytorch_lightning/callbacks/model_checkpoint.py Outdated Show resolved Hide resolved

less dangerous broadcast

d3fb41f

carmocca reviewed Sep 27, 2022

View reviewed changes

otaj and others added 3 commits September 27, 2022 18:37

fix mypy

a3ac6cd

Update src/pytorch_lightning/callbacks/model_checkpoint.py

edf2976

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

apply suggestions

7207946

carmocca reviewed Sep 27, 2022

View reviewed changes

Merge branch 'master' into bugfix/load_last_checkpoint

47066ac

mergify bot added the has conflicts label Sep 29, 2022

otaj and others added 3 commits September 30, 2022 14:20

merge master

b68e113

added test + small fix

d3a3b35

Merge branch 'master' into bugfix/load_last_checkpoint

cf95717

mergify bot removed the has conflicts label Sep 30, 2022

carmocca reviewed Sep 30, 2022

View reviewed changes

apply suggestions

ba3a48c

carmocca approved these changes Sep 30, 2022

View reviewed changes

mergify bot added the has conflicts label Sep 30, 2022

awaelchli approved these changes Sep 30, 2022

View reviewed changes

Merge branch 'master' into bugfix/load_last_checkpoint

fc20e62

awaelchli enabled auto-merge (squash) September 30, 2022 19:31

awaelchli disabled auto-merge September 30, 2022 19:31

awaelchli enabled auto-merge (squash) September 30, 2022 19:32

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Sep 30, 2022

awaelchli merged commit 511a070 into master Sep 30, 2022

awaelchli deleted the bugfix/load_last_checkpoint branch September 30, 2022 20:14

nicolai86 pushed a commit that referenced this pull request Oct 3, 2022

Find last checkpoints on restart (#14907)

e617a7b

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

nicolai86 pushed a commit that referenced this pull request Oct 13, 2022

Find last checkpoints on restart (#14907)

8f91655

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

nicolai86 pushed a commit that referenced this pull request Oct 13, 2022

Find last checkpoints on restart (#14907)

4ec9d63

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Find last checkpoints on restart #14907

Find last checkpoints on restart #14907

otaj commented Sep 27, 2022 •

edited by github-actions bot

Loading

carmocca left a comment

awaelchli commented Sep 27, 2022

otaj commented Sep 30, 2022

awaelchli left a comment

Find last checkpoints on restart #14907

Find last checkpoints on restart #14907

Conversation

otaj commented Sep 27, 2022 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Sep 27, 2022

otaj commented Sep 30, 2022

awaelchli left a comment

Choose a reason for hiding this comment

otaj commented Sep 27, 2022 •

edited by github-actions bot

Loading