Validate selected device indices in `DeepSpeedStrategy` #17952

awaelchli · 2023-06-30T10:16:13Z

What does this PR do?

In DeepSpeed, the assumption is that the model training happens on the device index that corresponds to the local rank:
https://github.com/microsoft/DeepSpeed/blob/fd1d2c64472c1a3061a05eb3b56a3f882199cfba/deepspeed/runtime/zero/partition_parameters.py#L841-L844

This means that the feature of the Trainer to select devices like Trainer(devices=[2, 5]) can't be supported. In this PR, I'm adding the validation and error message with the advice to set the CUDA_VISIBLE_DEVICES flag instead.

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda @awaelchli @carmocca @justusschock

for more information, see https://pre-commit.ci

…Lightning-AI/lightning into bugfix/deepspeed-device-index-selection

github-actions · 2023-07-01T02:15:30Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, tests/tests_fabric/strategies/test_deepspeed.py, tests/tests_fabric/strategies/test_deepspeed_integration.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, tests/tests_fabric/strategies/test_deepspeed.py, tests/tests_fabric/strategies/test_deepspeed_integration.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md, src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

tests/tests_pytorch/strategies/test_deepspeed_strategy.py

src/lightning/fabric/strategies/deepspeed.py

(cherry picked from commit 3f4790b)

awaelchli added 6 commits June 30, 2023 02:53

validate

349c0c3

x

45ef337

x

463719e

add test

93d7c71

change error

899ed87

add test

78940c0

awaelchli added bug Something isn't working strategy: deepspeed fun Staff contributions outside working hours - to differentiate from the "community" label labels Jun 30, 2023

awaelchli added this to the 2.0.x milestone Jun 30, 2023

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Jun 30, 2023

pre-commit-ci bot and others added 8 commits June 30, 2023 10:17

[pre-commit.ci] auto fixes from pre-commit.com hooks

73b72d5

for more information, see https://pre-commit.ci

remove test

fa3736b

Merge branch 'bugfix/deepspeed-device-index-selection' of github.com:…

f2427fc

…Lightning-AI/lightning into bugfix/deepspeed-device-index-selection

comment

1c6f9bf

import

0a33d97

changelog

d39023b

typing

c0bf4d4

delete test

949c6b1

awaelchli marked this pull request as ready for review July 1, 2023 02:14

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners July 1, 2023 02:14

awaelchli commented Jul 1, 2023

View reviewed changes

tests/tests_pytorch/strategies/test_deepspeed_strategy.py Show resolved Hide resolved

carmocca approved these changes Jul 1, 2023

View reviewed changes

src/lightning/fabric/strategies/deepspeed.py Show resolved Hide resolved

justusschock approved these changes Jul 3, 2023

View reviewed changes

mergify bot added ready PRs ready to be merged has conflicts labels Jul 3, 2023

mergify bot removed the ready PRs ready to be merged label Jul 3, 2023

Merge branch 'master' into bugfix/deepspeed-device-index-selection

4e90634

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jul 4, 2023

awaelchli enabled auto-merge (squash) July 4, 2023 14:11

awaelchli merged commit 3f4790b into master Jul 4, 2023
103 checks passed

awaelchli deleted the bugfix/deepspeed-device-index-selection branch July 4, 2023 18:58

Borda pushed a commit that referenced this pull request Jul 7, 2023

Validate selected device indices in DeepSpeedStrategy (#17952)

0d288f8

(cherry picked from commit 3f4790b)

Borda pushed a commit that referenced this pull request Jul 7, 2023

Validate selected device indices in DeepSpeedStrategy (#17952)

1e8e3a2

(cherry picked from commit 3f4790b)

lantiga pushed a commit that referenced this pull request Jul 10, 2023

Validate selected device indices in DeepSpeedStrategy (#17952)

3ba4ae7

(cherry picked from commit 3f4790b)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate selected device indices in `DeepSpeedStrategy` #17952

Validate selected device indices in `DeepSpeedStrategy` #17952

awaelchli commented Jun 30, 2023 •

edited by github-actions bot

github-actions bot commented Jul 1, 2023 •

edited

Validate selected device indices in DeepSpeedStrategy #17952

Validate selected device indices in DeepSpeedStrategy #17952

Conversation

awaelchli commented Jun 30, 2023 • edited by github-actions bot

What does this PR do?

PR review

github-actions bot commented Jul 1, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

Validate selected device indices in `DeepSpeedStrategy` #17952

Validate selected device indices in `DeepSpeedStrategy` #17952

awaelchli commented Jun 30, 2023 •

edited by github-actions bot

github-actions bot commented Jul 1, 2023 •

edited