Disable auto-detection of Kubeflow environment #18137

awaelchli · 2023-07-22T23:25:38Z

What does this PR do?

Several users have reported that they needed to unset the KUBERNETES_PORT environment variable in order to run multi-GPU training (#5254 and #16236). This is because the Trainer has auto-selected the Kubeflow environment despite the fact that the process wasn't launched using PyTorchJob. So the detection is unreliable.

I propose to disable the auto-detection. PyTorchJob users can still use the environment plugin by passing it to the trainer:

from lightning.pytorch.plugins.environments import KubeflowEnvironment

trainer = Trainer(plugins=KubeflowEnvironment())

Closes #16236

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda @justusschock @carmocca @awaelchli

src/lightning/fabric/plugins/environments/kubeflow.py

github-actions · 2023-07-22T23:29:17Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, tests/tests_fabric/plugins/environments/test_kubeflow.py, tests/tests_fabric/test_connector.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, tests/tests_fabric/plugins/environments/test_kubeflow.py, tests/tests_fabric/test_connector.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/environments/kubeflow.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md, src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

carmocca

This is a breaking change, no?

awaelchli · 2023-07-23T01:18:27Z

One could label it as such yes. We don't have any usage guides for it however.

(cherry picked from commit 41f0425)

Disable kubeflow auto-detection

cb65473

awaelchli added bug Something isn't working environment: kubeflow labels Jul 22, 2023

awaelchli added this to the 2.0.x milestone Jul 22, 2023

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Jul 22, 2023

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Jul 22, 2023

changelog

3f49dda

awaelchli commented Jul 22, 2023

View reviewed changes

src/lightning/fabric/plugins/environments/kubeflow.py Outdated Show resolved Hide resolved

awaelchli marked this pull request as ready for review July 22, 2023 23:28

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners July 22, 2023 23:28

carmocca approved these changes Jul 23, 2023

View reviewed changes

awaelchli added the breaking change Includes a breaking change label Jul 23, 2023

awaelchli added the ready PRs ready to be merged label Jul 26, 2023

awaelchli requested a review from Borda July 26, 2023 19:32

awaelchli added 2 commits July 27, 2023 02:11

review

3d3f72c

Merge branch 'master' into bugfix/kubeflow-autodetect

336be7a

mergify bot added has conflicts and removed ready PRs ready to be merged labels Jul 27, 2023

Borda approved these changes Jul 28, 2023

View reviewed changes

Merge branch 'master' into bugfix/kubeflow-autodetect

a5181f9

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jul 28, 2023

awaelchli merged commit 41f0425 into master Jul 28, 2023
104 checks passed

awaelchli deleted the bugfix/kubeflow-autodetect branch July 28, 2023 09:03

awaelchli mentioned this pull request Aug 14, 2023

Weekly minor patch release 2.0.7 [rebase & merge] #18270

Merged

7 tasks

Borda pushed a commit that referenced this pull request Aug 14, 2023

Disable auto-detection of Kubeflow environment (#18137)

0e8c349

(cherry picked from commit 41f0425)

lexierule pushed a commit that referenced this pull request Aug 14, 2023

Disable auto-detection of Kubeflow environment (#18137)

9cf52ec

(cherry picked from commit 41f0425)

This was referenced Aug 23, 2023

Launch ddp on 8 devices, but only run on the first gpu #16236

Closed

Multiple GPU per node could fail silently with KubeflowEnvironment #13374

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable auto-detection of Kubeflow environment #18137

Disable auto-detection of Kubeflow environment #18137

awaelchli commented Jul 22, 2023 •

edited by github-actions bot

github-actions bot commented Jul 22, 2023 •

edited

carmocca left a comment

awaelchli commented Jul 23, 2023

Disable auto-detection of Kubeflow environment #18137

Disable auto-detection of Kubeflow environment #18137

Conversation

awaelchli commented Jul 22, 2023 • edited by github-actions bot

What does this PR do?

PR review

github-actions bot commented Jul 22, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Jul 23, 2023

awaelchli commented Jul 22, 2023 •

edited by github-actions bot

github-actions bot commented Jul 22, 2023 •

edited