[profiler] Remove usage of onEachDevice from legacy profiler #54125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI [ghstack-poisoned]
ilia-cher
requested review from
albanD,
H-Huang,
mingzhe09088,
mrshenli,
pritamdamania87,
rohan-varma,
soulitzer and
zhaojuanmao
as code owners
facebook-github-bot
added
oncall: distributed
💊 CI failures summary and remediationsAs of commit 82d4433 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Differential Revision: [D27109481](https://our.internmc.facebook.com/intern/diff/D27109481) [ghstack-poisoned]
ilia-cher
added a commit
that referenced
this pull request
Mar 17, 2021
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI ghstack-source-id: 46680b804bc3a40e3cc3e7677b5608f12eb5d6ec Pull Request resolved: #54125
rohan-varma
reviewed
Mar 17, 2021
| if (it != startEvents.end()) { | ||
| e.setCudaUs(it->second->cudaElapsedUs(e)); | ||
| } else { | ||
| TORCH_WARN("Found a pop event without a corresponding push event"); |
There was a problem hiding this comment.
nit: might be good to log the event to help debugging. Also, should we setCudaUs to some reasonable value here?
There was a problem hiding this comment.
will set setCudaUs, also we don't have names in pop events
rohan-varma
approved these changes
Mar 17, 2021
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Differential Revision: [D27109481](https://our.internmc.facebook.com/intern/diff/D27109481) [ghstack-poisoned]
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Differential Revision: [D27109481](https://our.internmc.facebook.com/intern/diff/D27109481) [ghstack-poisoned]
|
fixed RPC test issue |
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Differential Revision: [D27109481](https://our.internmc.facebook.com/intern/diff/D27109481) [ghstack-poisoned]
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Differential Revision: [D27109481](https://our.internmc.facebook.com/intern/diff/D27109481) [ghstack-poisoned]
ilia-cher
added a commit
that referenced
this pull request
Mar 18, 2021
Summary: Fixes #48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI ghstack-source-id: 9a3444439bd0374456c9446133f0abfbdb15fc15 Pull Request resolved: #54125
Codecov Report
@@ Coverage Diff @@
## gh/ilia-cher/103/base #54125 +/- ##
=========================================================
- Coverage 77.47% 77.47% -0.01%
=========================================================
Files 1889 1889
Lines 184759 184732 -27
=========================================================
- Hits 143140 143117 -23
+ Misses 41619 41615 -4 |
|
@ilia-cher merged this pull request in 3b1e310. |
rohan-varma
added a commit
that referenced
this pull request
Apr 2, 2021
#54125 removed cuda event creation on each device from profiler, so #48987 should be resolved. To completely prevent the deadlock, we also make 2 further changes in the diff: 1) In LegacyProfiler, pass record_cuda=false for __stop_profile mark event because it is not a user op, and was resulting in cuda event being recorded on the wrong device (due to below reason) 2) Add torch.cuda.set_device() in tests to ensure that calls to `cudaGetDevice` in profiler don't return the wrong device. Note that using the profiler with `use_cuda=True` to profile distributed collectives isn't really an intended use case however, see the discussion in #52246. The profiling infrastructure has moved to primarily encourage the use of torch.profiler and CUPTI to trace CUDA kernels, support for distributed collectives for that will require further discussion with @ilia-cher, but we should still resolve this deadlock. Differential Revision: [D27491711](https://our.internmc.facebook.com/intern/diff/D27491711/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Apr 2, 2021
#54125 removed cuda event creation on each device from profiler, so #48987 should be resolved. To completely prevent the deadlock, we also make 2 further changes in the diff: 1) In LegacyProfiler, pass record_cuda=false for __stop_profile mark event because it is not a user op, and was resulting in cuda event being recorded on the wrong device (due to below reason) 2) Add torch.cuda.set_device() in tests to ensure that calls to `cudaGetDevice` in profiler don't return the wrong device. Note that using the profiler with `use_cuda=True` to profile distributed collectives isn't really an intended use case however, see the discussion in #52246. The profiling infrastructure has moved to primarily encourage the use of torch.profiler and CUPTI to trace CUDA kernels, support for distributed collectives for that will require further discussion with @ilia-cher, but we should still resolve this deadlock. Differential Revision: [D27491711](https://our.internmc.facebook.com/intern/diff/D27491711/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Apr 9, 2021
…is resolved" #54125 removed cuda event creation on each device from profiler, so #48987 should be resolved. To completely prevent the deadlock, we also make 2 further changes in the diff: 1) In LegacyProfiler, pass record_cuda=false for __stop_profile mark event because it is not a user op, and was resulting in cuda event being recorded on the wrong device (due to below reason) 2) Add torch.cuda.set_device() in tests to ensure that calls to `cudaGetDevice` in profiler don't return the wrong device. Note that using the profiler with `use_cuda=True` to profile distributed collectives isn't really an intended use case however, see the discussion in #52246. The profiling infrastructure has moved to primarily encourage the use of torch.profiler and CUPTI to trace CUDA kernels, support for distributed collectives for that will require further discussion with @ilia-cher, but we should still resolve this deadlock. Differential Revision: [D27491711](https://our.internmc.facebook.com/intern/diff/D27491711/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Apr 9, 2021
#54125 removed cuda event creation on each device from profiler, so #48987 should be resolved. To completely prevent the deadlock, we also make 2 further changes in the diff: 1) In LegacyProfiler, pass record_cuda=false for __stop_profile mark event because it is not a user op, and was resulting in cuda event being recorded on the wrong device (due to below reason) 2) Add torch.cuda.set_device() in tests to ensure that calls to `cudaGetDevice` in profiler don't return the wrong device. Note that using the profiler with `use_cuda=True` to profile distributed collectives isn't really an intended use case however, see the discussion in #52246. The profiling infrastructure has moved to primarily encourage the use of torch.profiler and CUPTI to trace CUDA kernels, support for distributed collectives for that will require further discussion with @ilia-cher, but we should still resolve this deadlock. Differential Revision: [D27491711](https://our.internmc.facebook.com/intern/diff/D27491711/) [ghstack-poisoned]
ilia-cher
changed the title
wubai
pushed a commit
to wubai/pytorch
that referenced
this pull request
Aug 7, 2023
Summary: Pull Request resolved: pytorch#54125 Fixes pytorch#48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Reviewed By: rohan-varma Differential Revision: D27109481 Pulled By: ilia-cher fbshipit-source-id: 3fba8bc55deafeed1ab4680b311e927f40eaf99c
wubai
pushed a commit
to wubai/pytorch
that referenced
this pull request
Aug 8, 2023
Summary: Pull Request resolved: pytorch#54125 Fixes pytorch#48987 Test Plan: python setup.py clean TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v python setup.py clean USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt python test/test_profiler.py -v + CI Reviewed By: rohan-varma Differential Revision: D27109481 Pulled By: ilia-cher fbshipit-source-id: 3fba8bc55deafeed1ab4680b311e927f40eaf99c
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary:
Fixes #48987
Test Plan:
python setup.py clean
TORCH_CUDA_ARCH_LIST="6.0" USE_CUDA=1 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt
python test/test_profiler.py -v
python setup.py clean
USE_CUDA=0 USE_MKLDNN=1 BLAS=MKL BUILD_BINARY=1 python setup.py develop install --cmake 2>&1 | tee ~/output.txt
python test/test_profiler.py -v
Differential Revision: D27109481