Add `Strategy.on_exception` #16646

awaelchli · 2023-02-06T13:49:26Z

What does this PR do?

Follow up to #16611
Closes #12445

This PR adds

def on_exception(self, exception):
    ...

to the Strategy base interface. An implementation can optionally override this method and process exceptions when they occur in Trainer. This will also be useful for #16410.

This makes it possible that the strategy can replace/edit an exception. For example, the following error emitted by DDP

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 0 1
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

gets replaced with a more useful one for Lightning users:

RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step. If this is intentional, you must enable the detection of unused parameters in DDP, either by setting the string value strategy='ddp_find_unused_parameters_true' or by setting the flag in the strategy with strategy=DDPStrategy(find_unused_parameters=True).

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @justusschock @awaelchli

for more information, see https://pre-commit.ci

…re/on-exception

for more information, see https://pre-commit.ci

tests/tests_pytorch/strategies/test_ddp_strategy.py

tests/tests_pytorch/strategies/test_ddp_spawn.py

tests/tests_pytorch/strategies/test_ddp_strategy.py

github-actions · 2023-02-06T15:11:27Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (slow, macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py, tests/tests_pytorch/strategies/test_ddp_spawn.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_exceptions.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py, tests/tests_pytorch/strategies/test_ddp_spawn.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_exceptions.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py, tests/tests_pytorch/strategies/test_ddp_spawn.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_exceptions.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py, tests/tests_pytorch/strategies/test_ddp_spawn.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_exceptions.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/trainer/call.py, src/lightning/pytorch/utilities/exceptions.py, src/lightning/pytorch/utilities/imports.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/pytorch/utilities/exceptions.py

src/lightning/pytorch/trainer/call.py

src/lightning/pytorch/utilities/exceptions.py

for more information, see https://pre-commit.ci

awaelchli added 4 commits February 6, 2023 14:23

on_exception

ba3c22d

fix

dbcd708

ddp spawn

7d87022

all

8c30aee

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 6, 2023

awaelchli added feature Is an improvement or enhancement strategy: ddp DistributedDataParallel labels Feb 6, 2023

awaelchli added this to the 2.0 milestone Feb 6, 2023

awaelchli and others added 12 commits February 6, 2023 14:52

import

d0629bc

[pre-commit.ci] auto fixes from pre-commit.com hooks

1db9979

for more information, see https://pre-commit.ci

spawn message

992411a

Merge remote-tracking branch 'origin/feature/on-exception' into featu…

f83e2c6

…re/on-exception

test

877b4aa

test

5ce0973

[pre-commit.ci] auto fixes from pre-commit.com hooks

51b51ae

for more information, see https://pre-commit.ci

add trainer test

d0aad76

test

66320f0

x

7218374

test

c770cbc

changelog

2384fd0

awaelchli commented Feb 6, 2023

View reviewed changes

tests/tests_pytorch/strategies/test_ddp_strategy.py Outdated Show resolved Hide resolved

awaelchli commented Feb 6, 2023

View reviewed changes

tests/tests_pytorch/strategies/test_ddp_spawn.py Outdated Show resolved Hide resolved

awaelchli commented Feb 6, 2023

View reviewed changes

tests/tests_pytorch/strategies/test_ddp_strategy.py Outdated Show resolved Hide resolved

awaelchli marked this pull request as ready for review February 6, 2023 15:10

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners February 6, 2023 15:10

carmocca reviewed Feb 6, 2023

View reviewed changes

src/lightning/pytorch/utilities/exceptions.py Outdated Show resolved Hide resolved

src/lightning/pytorch/trainer/call.py Show resolved Hide resolved

mergify bot added the has conflicts label Feb 6, 2023

Merge branch 'master' into feature/on-exception

ea0f084

mergify bot removed the has conflicts label Feb 6, 2023

awaelchli added 3 commits February 6, 2023 16:59

resolve todo

f6bda25

add note about PEP 678

57afa2f

also raise on keyboard interrupt

8f248c1

justusschock approved these changes Feb 7, 2023

View reviewed changes

carmocca approved these changes Feb 7, 2023

View reviewed changes

src/lightning/pytorch/utilities/exceptions.py Outdated Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Feb 7, 2023

awaelchli and others added 2 commits February 7, 2023 23:05

add python 3.11 support not sure why

1e82c21

[pre-commit.ci] auto fixes from pre-commit.com hooks

f0f9e37

for more information, see https://pre-commit.ci

awaelchli requested a review from carmocca February 7, 2023 22:07

Merge branch 'master' into feature/on-exception

f0c328d

awaelchli enabled auto-merge (squash) February 8, 2023 13:23

awaelchli merged commit 74ee699 into master Feb 8, 2023

awaelchli deleted the feature/on-exception branch February 8, 2023 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Strategy.on_exception` #16646

Add `Strategy.on_exception` #16646

awaelchli commented Feb 6, 2023 •

edited by github-actions bot

github-actions bot commented Feb 6, 2023 •

edited

Add Strategy.on_exception #16646

Add Strategy.on_exception #16646

Conversation

awaelchli commented Feb 6, 2023 • edited by github-actions bot

What does this PR do?

Before submitting

PR review

Did you have fun?

github-actions bot commented Feb 6, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

Add `Strategy.on_exception` #16646

Add `Strategy.on_exception` #16646

awaelchli commented Feb 6, 2023 •

edited by github-actions bot

github-actions bot commented Feb 6, 2023 •

edited