Avoid redundant input-type casting in FSDP precision #18630

awaelchli · 2023-09-25T11:14:58Z

What does this PR do?

Fixes #18612
This is the 2nd option from #18612 (comment)

FSDP converts inputs in its forward to the the param_type.

📚 Documentation preview 📚: https://pytorch-lightning--18630.org.readthedocs.build/en/18630/

cc @Borda @carmocca @justusschock @awaelchli

for more information, see https://pre-commit.ci

src/lightning/fabric/plugins/precision/fsdp.py

github-actions · 2023-09-25T20:45:24Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, src/lightning/pytorch/plugins/precision/fsdp.py, tests/tests_pytorch/plugins/precision/test_fsdp.py, tests/tests_pytorch/strategies/test_fsdp.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
[pytorch-lightning (GPUs) (testing Lightning	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=176007&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e)	success
[pytorch-lightning (GPUs) (testing PyTorch	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=176007&view=logs&jobId=3f274fac-2e11-54ca-487e-194c91f3ae9f)	success

These checks are required after the changes to src/lightning/pytorch/plugins/precision/fsdp.py, tests/tests_pytorch/plugins/precision/test_fsdp.py, tests/tests_pytorch/strategies/test_fsdp.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, src/lightning/pytorch/plugins/precision/fsdp.py.

🟢 fabric: Docs

Check ID	Status
docs-make (fabric, doctest)	success	✅
docs-make (fabric, html)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py.

🟢 pytorch_lightning: Docs

Check ID	Status
docs-make (pytorch, doctest)	success	✅
docs-make (pytorch, html)	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/precision/fsdp.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, tests/tests_fabric/plugins/precision/test_fsdp.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
[lightning-fabric (GPUs) (testing Fabric	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=176010&view=logs&jobId=3f274fac-2e11-54ca-487e-194c91f3ae9f)	success
[lightning-fabric (GPUs) (testing Lightning	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=176010&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e)	success

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, tests/tests_fabric/plugins/precision/test_fsdp.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, src/lightning/pytorch/plugins/precision/fsdp.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/fsdp.py, src/lightning/pytorch/plugins/precision/fsdp.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/fabric/CHANGELOG.md

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

codecov · 2023-09-26T11:53:24Z

Codecov Report

Merging #18630 (e2a1af0) into master (b542bbd) will decrease coverage by 31%.
Report is 5 commits behind head on master.
The diff coverage is 83%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18630      +/-   ##
==========================================
- Coverage      85%      55%     -31%     
==========================================
  Files         427      422       -5     
  Lines       33407    33341      -66     
==========================================
- Hits        28449    18221   -10228     
- Misses       4958    15120   +10162

Avoid redundant casting in FSDP precision

b817c4f

awaelchli added bug Something isn't working strategy: fsdp Fully Sharded Data Parallel labels Sep 25, 2023

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Sep 25, 2023

awaelchli added this to the 2.0.x milestone Sep 25, 2023

awaelchli and others added 11 commits September 25, 2023 13:16

chlog

474224d

change assertions

c2fd728

update test

332e2a0

update test

fa695d6

update tests

e426970

update

a2eda4a

update

157a6d7

update

0b65407

update pytorch

c758161

[pre-commit.ci] auto fixes from pre-commit.com hooks

bb8cb4d

for more information, see https://pre-commit.ci

more tests

5892648

awaelchli mentioned this pull request Sep 25, 2023

Remove outdated workaround for PyTorch autocast bug #18634

Merged

awaelchli commented Sep 25, 2023

View reviewed changes

src/lightning/fabric/plugins/precision/fsdp.py Show resolved Hide resolved

awaelchli marked this pull request as ready for review September 25, 2023 20:44

awaelchli requested review from carmocca, justusschock, Borda and williamFalcon as code owners September 25, 2023 20:44

carmocca approved these changes Sep 25, 2023

View reviewed changes

Borda approved these changes Sep 26, 2023

View reviewed changes

src/lightning/fabric/CHANGELOG.md Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Sep 26, 2023

Update src/lightning/fabric/CHANGELOG.md

e2a1af0

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

awaelchli merged commit 894952d into master Sep 26, 2023
113 of 115 checks passed

awaelchli deleted the bugfix/fsdp-redundant-cast branch September 26, 2023 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid redundant input-type casting in FSDP precision #18630

Avoid redundant input-type casting in FSDP precision #18630

awaelchli commented Sep 25, 2023 •

edited by github-actions bot

github-actions bot commented Sep 25, 2023 •

edited

codecov bot commented Sep 26, 2023 •

edited

Avoid redundant input-type casting in FSDP precision #18630

Avoid redundant input-type casting in FSDP precision #18630

Conversation

awaelchli commented Sep 25, 2023 • edited by github-actions bot

What does this PR do?

github-actions bot commented Sep 25, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Sep 26, 2023 • edited

Codecov Report

awaelchli commented Sep 25, 2023 •

edited by github-actions bot

github-actions bot commented Sep 25, 2023 •

edited

codecov bot commented Sep 26, 2023 •

edited