Fix device placement when `.cuda()` called without specifying index #14128

awaelchli · 2022-08-09T23:26:22Z

What does this PR do?

The PR #13704 resulted in an unintended change.

The second .cuda() call below should leave the model on the device set by torch.cuda.set_device if it is already on it.

model.cuda(1) 
torch.cuda.set_device(1)
model.cuda()  # On master, model lands on device 0. Expected is device 1

Reported on Slack.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @tchaton @rohitgr7 @carmocca @justusschock @awaelchli @Borda @ananthsub @ninginthecloud @jjenniferdai @akihironitta

nithinraok · 2022-08-10T00:02:58Z

Thanks this works, this is breaking change in 1.7.0. Would you be releasing minor version with this patch soon?

awaelchli · 2022-08-10T00:09:36Z

@nithinraok Yes, if merged, this would go into 1.7.2 next week (~Tuesday).

src/pytorch_lightning/core/mixins/device_dtype_mixin.py

codecov · 2022-08-10T02:20:15Z

Codecov Report

Merging #14128 (8cd35b1) into master (06c255c) will increase coverage by 15%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master   #14128     +/-   ##
=========================================
+ Coverage      61%      76%    +15%     
=========================================
  Files         324      324             
  Lines       26369    26493    +124     
=========================================
+ Hits        16138    20182   +4044     
+ Misses      10231     6311   -3920

…14128)

…ightning-AI#14128)

* update version and changelog for 1.7.2 release * Reset all results on epoch end (#14061) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> * Skip ddp fork tests on windows (#14121) * Fix device placement when `.cuda()` called without specifying index (#14128) * Convert subprocess test to standalone test (#14101) * Fix entry point test for Python 3.10 (#14154) * Fix flaky test caused by weak reference (#14157) * Fix saving hyperparameters in a composition where parent is not a LM or LDM (#14151) Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> * Remove DeepSpeed version restriction from Lite (#13967) * Configure the check-group app (#14165) Co-authored-by: Jirka <jirka.borovec@seznam.cz> * Update onnxruntime requirement from <=1.12.0 to <1.13.0 in /requirements (#14083) Updates the requirements on [onnxruntime](https://github.com/microsoft/onnxruntime) to permit the latest version. - [Release notes](https://github.com/microsoft/onnxruntime/releases) - [Changelog](https://github.com/microsoft/onnxruntime/blob/master/docs/ReleaseManagement.md) - [Commits](microsoft/onnxruntime@v0.1.4...v1.12.1) --- updated-dependencies: - dependency-name: onnxruntime dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update gcsfs requirement from <2022.6.0,>=2021.5.0 to >=2021.5.0,<2022.8.0 in /requirements (#14079) Update gcsfs requirement in /requirements Updates the requirements on [gcsfs](https://github.com/fsspec/gcsfs) to permit the latest version. - [Release notes](https://github.com/fsspec/gcsfs/releases) - [Commits](fsspec/gcsfs@2021.05.0...2022.7.1) --- updated-dependencies: - dependency-name: gcsfs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix a bug that caused spurious `AttributeError` when multiple `DataLoader` classes are imported (#14117) fix * CI: Replace `_` of in GHA workflow filenames with `-` (#13917) * Rename workflow files * Update docs * Fix azure badges * Update the main readme * bad rebase * Update doc * CI: Update Windows version from 2019 to 2022 (#14129) Update windows * CI/CD: Add CUDA version to docker image tags (#13831) * append cuda version to tags * revertme: push to hub * Update docker readme * Build base-conda-py3.9-torch1.12-cuda11.3.1 * Use new images in conda tests * revertme: push to hub * Revert "revertme: push to hub" This reverts commit 0f7d534. * Revert "revertme: push to hub" This reverts commit 46a05fc. * Run conda if workflow edited * Run gpu testing if workflow edited * Use new tags in release/Dockerfile * Build base-cuda and PL release images with all combinations * Update release docker * Update conda from py3.9-torch1.12 to py3.10-torch.1.12 * Fix ubuntu version * Revert conda * revertme: push to hub * Don't build Python 3.10 for now... * Fix pl release builder * updating version contribute to the error? docker/buildx#456 * Update actions' versions * Update slack user to notify * Don't use 11.6.0 to avoid bagua incompatibility * Don't use 11.1, and use 11.1.1 * Update .github/workflows/ci-pytorch_test-conda.yml Co-authored-by: Luca Medeiros <67411094+luca-medeiros@users.noreply.github.com> * Update trigger * Ignore artfacts from tutorials * Trim docker images to distribute * Add an image for tutorials * Update conda image 3.8x1.10 * Try different conda variants * No need to set cuda for conda jobs * Update who to notify ipu failure * Don't push * update filenaem Co-authored-by: Luca Medeiros <67411094+luca-medeiros@users.noreply.github.com> * Avoid entry_points deprecation warning (#14052) Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> * Configure the check-group app (#14165) Co-authored-by: Jirka <jirka.borovec@seznam.cz> * Profile batch transfer and gradient clipping hooks (#14069) Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> * Avoid false positive warning about using `sync_dist` when using torchmetrics (#14143) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> * Avoid raising the sampler warning if num_replicas=1 (#14097) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> Co-authored-by: otaj <6065855+otaj@users.noreply.github.com> * Remove skipping logic in favor of path filtering (#14170) * Support checkpoint save and load with Stochastic Weight Averaging (#9938) Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Carlos Mocholi <carlossmocholi@gmail.com> Co-authored-by: Kushashwa Ravi Shrimali <kushashwaravishrimali@gmail.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> * Use fsdp module to initialize precision scalar for fsdp native (#14092) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Laverne Henderson <laverne.henderson@coupa.com> Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> * add more issues types (#14174) * add more issues types * Update .github/ISSUE_TEMPLATE/config.yml Co-authored-by: Mansy <ahmed.mansy156@gmail.com> * typo Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com> Co-authored-by: Mansy <ahmed.mansy156@gmail.com> Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Laverne Henderson <laverne.henderson@coupa.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> * CI: clean building docs (#14216) * CI: clean building docs * group * . * CI: docker focus on PL only (#14246) * CI: docker focus on PL only * group * Allowed setting attributes on `DataLoader` and `BatchSampler` when instantiated inside `*_dataloader` hooks (#14212) Co-authored-by: otaj <6065855+otaj@users.noreply.github.com> * Revert "Remove skipping logic in favor of path filtering (#14170)" (#14244) * Update defaults for WandbLogger's run name and project name (#14145) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> Co-authored-by: Luca Medeiros <67411094+luca-medeiros@users.noreply.github.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: otaj <6065855+otaj@users.noreply.github.com> Co-authored-by: Adam Reeve <adreeve@gmail.com> Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Kushashwa Ravi Shrimali <kushashwaravishrimali@gmail.com> Co-authored-by: Laverne Henderson <laverne.henderson@coupa.com> Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com> Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com> Co-authored-by: Mansy <ahmed.mansy156@gmail.com>

awaelchli added 2 commits August 10, 2022 01:20

fix device placement with cuda call

983c3b4

update

bc87791

github-actions bot added the pl Generic label for PyTorch Lightning package label Aug 9, 2022

awaelchli changed the title ~~Bugfix/cuda~~ Fix device placement when .cuda() called without specifying index Aug 9, 2022

update changelog

8cd35b1

awaelchli self-assigned this Aug 9, 2022

awaelchli added bug Something isn't working priority: 0 High priority task lightningmodule pl.LightningModule labels Aug 9, 2022

awaelchli added this to the pl:1.7.x milestone Aug 9, 2022

awaelchli marked this pull request as ready for review August 10, 2022 00:08

awaelchli requested review from tchaton, Borda, carmocca, justusschock, kaushikb11, williamFalcon and rohitgr7 as code owners August 10, 2022 00:08

carmocca approved these changes Aug 10, 2022

View reviewed changes

src/pytorch_lightning/core/mixins/device_dtype_mixin.py Show resolved Hide resolved

akihironitta approved these changes Aug 10, 2022

View reviewed changes

mergify bot added the ready PRs ready to be merged label Aug 10, 2022

Borda approved these changes Aug 10, 2022

View reviewed changes

rohitgr7 approved these changes Aug 10, 2022

View reviewed changes

kaushikb11 approved these changes Aug 10, 2022

View reviewed changes

justusschock approved these changes Aug 10, 2022

View reviewed changes

awaelchli merged commit dc8ff5e into master Aug 10, 2022

awaelchli deleted the bugfix/cuda branch August 10, 2022 09:23

awaelchli added a commit that referenced this pull request Aug 10, 2022

Fix device placement when .cuda() called without specifying index (#…

c76c381

…14128)

nithinraok mentioned this pull request Aug 10, 2022

upgrade to PTL 1.7 NVIDIA/NeMo#4672

Merged

9 tasks

jessecambon pushed a commit to jessecambon/lightning that referenced this pull request Aug 16, 2022

Fix device placement when .cuda() called without specifying index (L…

1f07ea4

…ightning-AI#14128)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix device placement when `.cuda()` called without specifying index #14128

Fix device placement when `.cuda()` called without specifying index #14128

awaelchli commented Aug 9, 2022 •

edited by github-actions bot

nithinraok commented Aug 10, 2022

awaelchli commented Aug 10, 2022

codecov bot commented Aug 10, 2022

Fix device placement when .cuda() called without specifying index #14128

Fix device placement when .cuda() called without specifying index #14128

Conversation

awaelchli commented Aug 9, 2022 • edited by github-actions bot

What does this PR do?

Before submitting

PR review

Did you have fun?

nithinraok commented Aug 10, 2022

awaelchli commented Aug 10, 2022

codecov bot commented Aug 10, 2022

Codecov Report

Fix device placement when `.cuda()` called without specifying index #14128

Fix device placement when `.cuda()` called without specifying index #14128

awaelchli commented Aug 9, 2022 •

edited by github-actions bot