Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix comm initialization in MPIEnvironment #19074

Merged
merged 5 commits into from Nov 28, 2023
Merged

Fix comm initialization in MPIEnvironment #19074

merged 5 commits into from Nov 28, 2023

Conversation

awaelchli
Copy link
Member

@awaelchli awaelchli commented Nov 27, 2023 •
edited by github-actions bot

@github-actions github-actions bot added the fabric lightning.fabric.Fabric label Nov 27, 2023
@awaelchli awaelchli added bug Something isn't working fun Staff contributions outside working hours - to differentiate from the "community" label environment: mpi labels Nov 27, 2023
@awaelchli awaelchli added this to the 2.1.x milestone Nov 27, 2023
@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Nov 27, 2023
@awaelchli awaelchli marked this pull request as ready for review November 27, 2023 23:51
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 27, 2023 •
edited

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success ✅
pl-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
pl-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
pl-cpu (macOS-11, lightning, 3.10, 2.0) success ✅
pl-cpu (macOS-11, lightning, 3.10, 2.1) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success ✅
pl-cpu (windows-2022, lightning, 3.9, 1.12) success ✅
pl-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.10, 2.0) success ✅
pl-cpu (windows-2022, lightning, 3.10, 2.1) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success ✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0) success ✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1) success ✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success ✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1) success ✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success ✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) (testing Lightning | latest) success ✅
pytorch-lightning (GPUs) (testing PyTorch | latest) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

🟢 pytorch_lightning: Benchmarks
Check ID Status
lightning.Benchmarks success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

🟢 fabric: Docs
Check ID Status
docs-make (fabric, doctest) success ✅
docs-make (fabric, html) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success ✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0) success ✅
fabric-cpu (macOS-11, lightning, 3.11, 2.1) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success ✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0) success ✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1) success ✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success ✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success ✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0) success ✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1) success ✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0) success ✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1) success ✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0) success ✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py, tests/tests_fabric/plugins/environments/test_mpi.py, .github/workflows/ci-tests-fabric.yml.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) (testing Fabric | latest) success ✅
lightning-fabric (GPUs) (testing Lightning | latest) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py, tests/tests_fabric/plugins/environments/test_mpi.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success ✅
install-pkg (ubuntu-22.04, app, 3.11) success ✅
install-pkg (ubuntu-22.04, fabric, 3.8) success ✅
install-pkg (ubuntu-22.04, fabric, 3.11) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.8) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.11) success ✅
install-pkg (ubuntu-22.04, lightning, 3.8) success ✅
install-pkg (ubuntu-22.04, lightning, 3.11) success ✅
install-pkg (ubuntu-22.04, notset, 3.8) success ✅
install-pkg (ubuntu-22.04, notset, 3.11) success ✅
install-pkg (macOS-12, app, 3.8) success ✅
install-pkg (macOS-12, app, 3.11) success ✅
install-pkg (macOS-12, fabric, 3.8) success ✅
install-pkg (macOS-12, fabric, 3.11) success ✅
install-pkg (macOS-12, pytorch, 3.8) success ✅
install-pkg (macOS-12, pytorch, 3.11) success ✅
install-pkg (macOS-12, lightning, 3.8) success ✅
install-pkg (macOS-12, lightning, 3.11) success ✅
install-pkg (macOS-12, notset, 3.8) success ✅
install-pkg (macOS-12, notset, 3.11) success ✅
install-pkg (windows-2022, app, 3.8) success ✅
install-pkg (windows-2022, app, 3.11) success ✅
install-pkg (windows-2022, fabric, 3.8) success ✅
install-pkg (windows-2022, fabric, 3.11) success ✅
install-pkg (windows-2022, pytorch, 3.8) success ✅
install-pkg (windows-2022, pytorch, 3.11) success ✅
install-pkg (windows-2022, lightning, 3.8) success ✅
install-pkg (windows-2022, lightning, 3.11) success ✅
install-pkg (windows-2022, notset, 3.8) success ✅
install-pkg (windows-2022, notset, 3.11) success ✅

These checks are required after the changes to src/lightning/fabric/plugins/environments/mpi.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@codecov Codecov
Copy link

codecov bot commented Nov 28, 2023 •
edited

Codecov Report

Merging #19074 (0dc1df8) into master (9d36604) will decrease coverage by 28%.
The diff coverage is 100%.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #19074      +/-   ##
==========================================
- Coverage      82%      54%     -28%     
==========================================
  Files         443      438       -5     
  Lines       36612    36514      -98     
==========================================
- Hits        30090    19710   -10380     
- Misses       6522    16804   +10282

@github-actions github-actions bot added the ci Continuous Integration label Nov 28, 2023
@mergify mergify bot added the ready PRs ready to be merged label Nov 28, 2023
@awaelchli awaelchli merged commit 197b225 into master Nov 28, 2023
119 checks passed
@awaelchli awaelchli deleted the bugfix/mpi-broadcast branch November 28, 2023 21:14
@awaelchli awaelchli mentioned this pull request Nov 29, 2023
Closed
Borda pushed a commit that referenced this pull request Dec 19, 2023
@awaelchli @Borda
Fix comm initialization in MPIEnvironment (#19074)
e1356ed 
(cherry picked from commit 197b225)
lantiga pushed a commit that referenced this pull request Dec 20, 2023
@awaelchli @lantiga
Fix comm initialization in MPIEnvironment (#19074)
6e52562 
(cherry picked from commit 197b225)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Borda Borda Borda approved these changes

@carmocca carmocca carmocca approved these changes

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@williamFalcon williamFalcon Awaiting requested review from williamFalcon williamFalcon is a code owner

@ethanwharris ethanwharris Awaiting requested review from ethanwharris ethanwharris is a code owner

Assignees
No one assigned
Labels
bug Something isn't working ci Continuous Integration environment: mpi fabric lightning.fabric.Fabric fun Staff contributions outside working hours - to differentiate from the "community" label pl Generic label for PyTorch Lightning package ready PRs ready to be merged
Projects
None yet
Milestone
2.1.x
Development

Successfully merging this pull request may close these issues.

MPIEnvironment fails for MPI multi-node training at comm gather step of worker nodes
3 participants
@awaelchli @Borda @carmocca