Fix state dict loading for ddp/dp in Fabric #17997

awaelchli · 2023-07-06T09:26:47Z

What does this PR do?

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda @tchaton @carmocca @justusschock @awaelchli

for more information, see https://pre-commit.ci

… bugfix/load-state-dict-ddp

github-actions · 2023-07-06T09:31:19Z

⛈️ Required checks status: Has failure 🔴

Warning
This job will need to be re-run to merge your PR. If you do not have write access to the repository, you can ask Lightning-AI/lai-frameworks to re-run it. If you push a new commit, all of CI will re-trigger.

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py, tests/tests_fabric/strategies/test_ddp.py, tests/tests_fabric/strategies/test_ddp_integration.py, tests/tests_fabric/strategies/test_strategy.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py, tests/tests_fabric/strategies/test_ddp.py, tests/tests_fabric/strategies/test_ddp_integration.py, tests/tests_fabric/strategies/test_strategy.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/dp.py, src/lightning/fabric/strategies/strategy.py.

🔴 link-check

Check ID	Status
check-md-links / markdown-link-check	failure	❌

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

* fix state dict loading for ddp/dp * test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changelog * update test * move params to same device before equality test * test strategy --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit b14ddd9)

awaelchli added 2 commits July 6, 2023 10:55

fix state dict loading for ddp/dp

abf976a

test

89969fa

awaelchli added bug Something isn't working fabric lightning.fabric.Fabric strategy: ddp DistributedDataParallel labels Jul 6, 2023

awaelchli added this to the 2.0.x milestone Jul 6, 2023

awaelchli self-assigned this Jul 6, 2023

awaelchli added the priority: 0 High priority task label Jul 6, 2023

pre-commit-ci bot and others added 3 commits July 6, 2023 09:27

[pre-commit.ci] auto fixes from pre-commit.com hooks

0aae8fa

for more information, see https://pre-commit.ci

changelog

ce92593

Merge remote-tracking branch 'origin/bugfix/load-state-dict-ddp' into…

311fd75

… bugfix/load-state-dict-ddp

awaelchli marked this pull request as ready for review July 6, 2023 09:30

awaelchli requested review from carmocca and justusschock as code owners July 6, 2023 09:30

awaelchli added 3 commits July 6, 2023 12:09

update test

53bfd29

move params to same device before equality test

2aa4a29

test strategy

ccb128a

carmocca approved these changes Jul 6, 2023

View reviewed changes

justusschock approved these changes Jul 6, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Jul 6, 2023

lantiga merged commit b14ddd9 into master Jul 6, 2023
100 of 102 checks passed

lantiga deleted the bugfix/load-state-dict-ddp branch July 6, 2023 11:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix state dict loading for ddp/dp in Fabric #17997

Fix state dict loading for ddp/dp in Fabric #17997

awaelchli commented Jul 6, 2023 •

edited by github-actions bot

github-actions bot commented Jul 6, 2023 •

edited

Fix state dict loading for ddp/dp in Fabric #17997

Fix state dict loading for ddp/dp in Fabric #17997

Conversation

awaelchli commented Jul 6, 2023 • edited by github-actions bot

What does this PR do?

PR review

github-actions bot commented Jul 6, 2023 • edited

⛈️ Required checks status: Has failure 🔴

Groups summary

awaelchli commented Jul 6, 2023 •

edited by github-actions bot

github-actions bot commented Jul 6, 2023 •

edited