Eliminate device guard in generic dispatch key kernel wrappers #55131

wenleix · 2021-04-01T06:21:58Z

Stack from ghstack:

[RFC] Unify device check and guard in kernel wrapper #55241 [RFC] Unify device check and guard in kernel wrapper
Eliminate device guard in generic dispatch key kernel wrappers #55131 Eliminate device guard in generic dispatch key kernel wrappers

Benchmark zeros_out:

from torch.utils.benchmark import Timer
counts = Timer(
    stmt="""at::zeros_out(t, {1});""",
    setup="auto t = at::empty({1});",
    language="cpp",
).collect_callgrind(number=1_000)
print(counts)

With device guard:

<torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0>
at::zeros_out(t, {1});
setup: auto t = at::empty({1});
                           All          Noisy symbols removed
    Instructions:      1396022                    1396022
    Baseline:                0                          0
1000 runs per measurement, 1 thread

Without device guard:

<torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0>
at::zeros_out(t, {1});
setup: auto t = at::empty({1});
                           All          Noisy symbols removed
    Instructions:      1296022                    1296022
    Baseline:                0                          0
1000 runs per measurement, 1 thread

We see about 7.7% improvement.

Differential Revision: D27496584

Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

facebook-github-bot · 2021-04-01T06:22:06Z

💊 CI failures summary and remediations

As of commit a7bc765 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-scanned failure(s)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) ghstack-source-id: 125477236 Pull Request resolved: #55131

wenleix · 2021-04-01T16:41:08Z

This solves #47778

…y kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

Pull Request resolved: #55131 ghstack-source-id: 125527889 Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/)

… kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…el wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

Pull Request resolved: #55131 ghstack-source-id: 125594203 Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/)

…c dispatch key kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…ernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…pers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…kernel wrappers" Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…n generic dispatch key kernel wrappers" Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…ispatch key kernel wrappers" Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

… kernel wrappers" Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

…pers" Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

facebook-github-bot · 2021-04-12T22:44:05Z

This pull request has been merged in 561b507.

…ch#55131) Summary: Pull Request resolved: pytorch#55131 Benchmark `zeros_out`: ```python from torch.utils.benchmark import Timer counts = Timer( stmt="""at::zeros_out(t, {1});""", setup="auto t = at::empty({1});", language="cpp", ).collect_callgrind(number=1_000) print(counts) ``` With device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f834f095ca0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1396022 1396022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` Without device guard: ``` <torch.utils.benchmark.utils.valgrind_wrapper.timer_interface.CallgrindStats object at 0x7f25e48927c0> at::zeros_out(t, {1}); setup: auto t = at::empty({1}); All Noisy symbols removed Instructions: 1296022 1296022 Baseline: 0 0 1000 runs per measurement, 1 thread ``` We see about `7.7%` improvement. ghstack-source-id: 126295368 Test Plan: ``` buck build //caffe2/aten/... buck test mode/dev mode/no-gpu //caffe2/test:torch -- 'caffe2/test:torch - test_msnpu_error (test_torch.TestTorch)' ``` Reviewed By: ezyang Differential Revision: D27496584 fbshipit-source-id: 97f783a809b77b28f77a93096d69b3da9ee69df7

Eliminate device guard in generic dispatch key kernel wrappers

443c337

Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

facebook-github-bot added the cla signed label Apr 1, 2021

wenleix mentioned this pull request Apr 1, 2021

Simplify device guard code generation #55112

Closed

wenleix pushed a commit that referenced this pull request Apr 1, 2021

Eliminate device guard in generic dispatch key kernel wrappers

effc4db

Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) ghstack-source-id: 125477236 Pull Request resolved: #55131

rebase; minor tweak on "Eliminate device guard in generic dispatch ke…

745ec22

…y kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

wenleix pushed a commit that referenced this pull request Apr 1, 2021

Eliminate device guard in generic dispatch key kernel wrappers

27709ec

Pull Request resolved: #55131 ghstack-source-id: 125527889 Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/)

wenleix marked this pull request as draft April 1, 2021 18:56

add some assertion on "Eliminate device guard in generic dispatch key…

dd83b07

… kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

try some hack on "Eliminate device guard in generic dispatch key kern…

ee051f2

…el wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

wenleix pushed a commit that referenced this pull request Apr 2, 2021

Eliminate device guard in generic dispatch key kernel wrappers

060305f

Pull Request resolved: #55131 ghstack-source-id: 125594203 Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/)

wenleix mentioned this pull request Apr 2, 2021

[RFC] Unify device check and guard in kernel wrapper #55241

Closed

wxie added 2 commits April 3, 2021 17:07

rebase; more breakpoint friendly on "Eliminate device guard in generi…

5bbf9dd

…c dispatch key kernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

rebase to latest on "Eliminate device guard in generic dispatch key k…

3854f8c

…ernel wrappers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

wenleix marked this pull request as ready for review April 5, 2021 17:00

wenleix requested review from ezyang and bhosmer April 5, 2021 17:03

wxie added 3 commits April 5, 2021 10:19

rebase on "Eliminate device guard in generic dispatch key kernel wrap…

2c93d52

…pers" Differential Revision: [D27496584](https://our.internmc.facebook.com/intern/diff/D27496584/) [ghstack-poisoned]

ezyang approved these changes Apr 8, 2021

View reviewed changes

wxie added 3 commits April 7, 2021 21:11

facebook-github-bot closed this in 561b507 Apr 12, 2021

facebook-github-bot added the Merged label Apr 12, 2021

facebook-github-bot deleted the gh/wenleix/16/head branch April 16, 2021 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Eliminate device guard in generic dispatch key kernel wrappers #55131

Eliminate device guard in generic dispatch key kernel wrappers #55131

Uh oh!

wenleix commented Apr 1, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Apr 1, 2021 •

edited

Loading

Uh oh!

wenleix commented Apr 1, 2021

Uh oh!

facebook-github-bot commented Apr 12, 2021

Uh oh!

Uh oh!

Eliminate device guard in generic dispatch key kernel wrappers #55131

Eliminate device guard in generic dispatch key kernel wrappers #55131

Uh oh!

Conversation

wenleix commented Apr 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Apr 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

wenleix commented Apr 1, 2021

Uh oh!

facebook-github-bot commented Apr 12, 2021

Uh oh!

Uh oh!

wenleix commented Apr 1, 2021 •

edited

Loading

facebook-github-bot commented Apr 1, 2021 •

edited

Loading