Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fabric: drop FairScale's sharded implementation #16329

Merged
merged 15 commits into from
Jan 11, 2023
Merged

Fabric: drop FairScale's sharded implementation #16329

merged 15 commits into from
Jan 11, 2023

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Jan 10, 2023 •
edited by github-actions bot

What does this PR do?

Removes strategy='ddp_sharded'|'ddp_sharded_spawn' from LightningLite and Fabric. FSDP is the recommended replacement.

For the closest alternative, Fabric(strategy=FSDPStrategy(sharding_strategy=ShardingStrategy.SHARD_GRAD_OP)) is suggested.

Does your PR introduce any breaking changes? If yes, please list them.

Removes Fabrics and LightningLite FairScale integration

cc @Borda @justusschock @carmocca @awaelchli

@carmocca carmocca added deprecation Includes a deprecation breaking change Includes a breaking change fabric lightning.fabric.Fabric strategy: fairscale fsdp (removed) Fully Sharded Data Parallel labels Jan 10, 2023
@carmocca carmocca added this to the v1.9 milestone Jan 10, 2023
@carmocca carmocca self-assigned this Jan 10, 2023
@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Jan 10, 2023
@carmocca carmocca force-pushed the fabric/replace-fairscale-fsdp branch from 77157f4 to 900370f Compare January 10, 2023 22:20
@carmocca carmocca changed the title Fabric: replace FairScale's sharded implementation with FSDP Fabric: drop FairScale's sharded implementation Jan 11, 2023
@carmocca carmocca added strategy: fairscale sharded (removed) Sharded Data Parallel and removed deprecation Includes a deprecation strategy: fairscale fsdp (removed) Fully Sharded Data Parallel labels Jan 11, 2023
@carmocca carmocca marked this pull request as ready for review January 11, 2023 13:36
@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2023 •
edited

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11) success ✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12) success ✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest) success ✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11) success ✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12) success ✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest) success ✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11) success ✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11) success ✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11) success ✅
pl-cpu (macOS-11, lightning, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.13) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success ✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success ✅
make-html (pytorch) success ✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, docs/source-pytorch/fabric/api/api_reference.rst, docs/source-pytorch/fabric/api/fabric_args.rst.

🟢 pytorch_lightning: Docker
Check ID Status
build-cuda (3.9, 1.10, 11.3.1) success ✅
build-cuda (3.9, 1.11, 11.3.1) success ✅
build-cuda (3.9, 1.12, 11.6.1) success ✅
build-cuda (3.9, 1.13, 11.7.1) success ✅
build-hpu (1.5.0, 1.11.0) success ✅
build-ipu (3.9, 1.10) success ✅
build-NGC success ✅
build-pl (3.9, 1.10, 11.3.1) success ✅
build-pl (3.9, 1.11, 11.3.1) success ✅
build-pl (3.9, 1.12, 11.6.1) success ✅
build-pl (3.9, 1.13, 11.7.1) success ✅
build-xla (3.7, 1.12) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, fabric, 3.8, 1.11) success ✅
fabric-cpu (macOS-11, fabric, 3.9, 1.12) success ✅
fabric-cpu (macOS-11, fabric, 3.10, 1.13) success ✅
fabric-cpu (macOS-11, fabric, 3.7, 1.10, oldest) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.10) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.9, 1.11) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.12) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.13) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.7, 1.10, oldest) success ✅
fabric-cpu (windows-2022, fabric, 3.9, 1.11) success ✅
fabric-cpu (windows-2022, fabric, 3.10, 1.12) success ✅
fabric-cpu (windows-2022, fabric, 3.10, 1.13) success ✅
fabric-cpu (windows-2022, fabric, 3.7, 1.10, oldest) success ✅
fabric-cpu (macOS-11, lightning, 3.8, 1.13) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, tests/tests_fabric/helpers/models.py, tests/tests_fabric/helpers/runif.py, tests/tests_fabric/strategies/test_fairscale.py, tests/tests_fabric/strategies/test_fairscale_integration.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_registry.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_fabric.py, tests/tests_fabric/utilities/test_imports.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, tests/tests_fabric/helpers/models.py, tests/tests_fabric/helpers/runif.py, tests/tests_fabric/strategies/test_fairscale.py, tests/tests_fabric/strategies/test_fairscale_integration.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_registry.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_fabric.py, tests/tests_fabric/utilities/test_imports.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.7) success ✅
install-pkg (ubuntu-22.04, app, 3.10) success ✅
install-pkg (ubuntu-22.04, fabric, 3.7) success ✅
install-pkg (ubuntu-22.04, fabric, 3.10) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.7) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.10) success ✅
install-pkg (ubuntu-22.04, lightning, 3.7) success ✅
install-pkg (ubuntu-22.04, lightning, 3.10) success ✅
install-pkg (ubuntu-22.04, notset, 3.7) success ✅
install-pkg (ubuntu-22.04, notset, 3.10) success ✅
install-pkg (macOS-12, app, 3.7) success ✅
install-pkg (macOS-12, app, 3.10) success ✅
install-pkg (macOS-12, fabric, 3.7) success ✅
install-pkg (macOS-12, fabric, 3.10) success ✅
install-pkg (macOS-12, pytorch, 3.7) success ✅
install-pkg (macOS-12, pytorch, 3.10) success ✅
install-pkg (macOS-12, lightning, 3.7) success ✅
install-pkg (macOS-12, lightning, 3.10) success ✅
install-pkg (macOS-12, notset, 3.7) success ✅
install-pkg (macOS-12, notset, 3.10) success ✅
install-pkg (windows-2022, app, 3.7) success ✅
install-pkg (windows-2022, app, 3.10) success ✅
install-pkg (windows-2022, fabric, 3.7) success ✅
install-pkg (windows-2022, fabric, 3.10) success ✅
install-pkg (windows-2022, pytorch, 3.7) success ✅
install-pkg (windows-2022, pytorch, 3.10) success ✅
install-pkg (windows-2022, lightning, 3.7) success ✅
install-pkg (windows-2022, lightning, 3.10) success ✅
install-pkg (windows-2022, notset, 3.7) success ✅
install-pkg (windows-2022, notset, 3.10) success ✅

These checks are required after the changes to src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, requirements/fabric/strategies.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

awaelchli
awaelchli approved these changes Jan 11, 2023
View reviewed changes
src/pytorch_lightning/lite/lite.py Outdated Show resolved Hide resolved
tests/tests_fabric/strategies/test_fsdp_integration.py Show resolved Hide resolved
tests/tests_fabric/helpers/models.py Show resolved Hide resolved
@mergify mergify bot added the ready PRs ready to be merged label Jan 11, 2023
@carmocca @awaelchli
Apply suggestions from code review
45735d1 
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
@carmocca carmocca enabled auto-merge (squash) January 11, 2023 14:50
@mergify mergify bot added has conflicts and removed ready PRs ready to be merged labels Jan 11, 2023
@mergify mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jan 11, 2023
@carmocca carmocca merged commit 428844d into master Jan 11, 2023
@carmocca carmocca deleted the fabric/replace-fairscale-fsdp branch January 11, 2023 17:08
@Borda Borda mentioned this pull request Jan 12, 2023
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awaelchli awaelchli awaelchli approved these changes

@justusschock justusschock justusschock approved these changes

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@edenlightning edenlightning Awaiting requested review from edenlightning

@tchaton tchaton Awaiting requested review from tchaton tchaton is a code owner

@Borda Borda Awaiting requested review from Borda Borda is a code owner

Assignees

@carmocca carmocca

Labels
breaking change Includes a breaking change fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package ready PRs ready to be merged strategy: fairscale sharded (removed) Sharded Data Parallel
Projects
None yet
Milestone
v1.9
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@carmocca @awaelchli @justusschock