[PyTorch][Vulkan] Fix matrix multiplication performance test binary #114624

copyrightly · 2023-11-27T20:42:27Z

Summary:
Due to recent changes in D51421256 and D51379737,

shaders of mm, addmm, bmm, baddbmm are reduced into just mm,
height and width packing logic is applied to linear operations

so the current perf testings of addmm and create_linear_context and run_linear_context are no longer valid (0 latency will be printed, see test plan). Specifically, the original test extracts latency of vulkan.addmm which doesn't exist any more. Instead the current implementation of addmm invokes

vulkan.convert_channels_to_height_packed
vulkan.convert_channels_to_width_packed
vulkan.mm
vulkan.mul_scalar
vulkan.add

To deal with this

for addmm and run_linear_context, we apply a new function extractTotalShaderResultsAndSetState which aggregates latency of all invoded shaders except nchw_to_image and image_to_nchw;
for create_linear_context, besides nchw_to_image and image_to_nchw, we also aggregate vulkan.convert_channels_to_height_packed

Test Plan:

build binary, at fbsource

buck2 build  -c ndk.debug_info_level=0  -c ndk.static_linking=true -c pt.enable_qpl=0 -c pt.vulkan_use_gpu_diagnostics=1 --target-platforms=ovr_config//platform/android:arm32-fbsource //xplat/caffe2:pt_vulkan_mm_perf_test_binAndroid  --show-output  -c pt.vulkan_full_precision=1

test on android device

adb push buck-out/v2/gen/fbsource/f1f3f9bed27e143c/xplat/caffe2/__pt_vulkan_mm_perf_test_binAndroid__/pt_vulkan_mm_perf_test_binAndroid /data/local/tmp
adb shell /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid

Before

addmm_benchmark

(base) luwei@luwei-mbp ~ % adb shell /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid
2023-11-16T06:48:18+00:00
Running /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid
Run on (4 X 1708.8 MHz CPU s)
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
...
Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4334408
vulkan.nchw_to_image     {500, 500, 1}                    4327648
vulkan.nchw_to_image     {500, 500, 1}                    4322760
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1233960
vulkan.convert_channels_to_width_packed{125, 500, 1}                    1286896
vulkan.mm                {125, 125, 1}                   76186084
vulkan.mul_scalar        {500, 500, 1}                    1132924
vulkan.mul_scalar        {500, 500, 1}                    1128556
vulkan.add               {500, 500, 1}                    4285788
vulkan.image_to_nchw     {500, 500, 1}                    1421576
...
addmm_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1                      0.000 ms         77.2 ms            5

create_linear_context_benchmark

Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4336696
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1229384
...
create_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1       8.57 ms         32.9 ms            5

run_linear_context_benchmark

Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4305548
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1196104
...
run_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1         0.000 ms         86.2 ms            5

After

addmm_benchmark

Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4332016
vulkan.nchw_to_image     {500, 500, 1}                    4321356
vulkan.nchw_to_image     {500, 500, 1}                    4314908
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1195896
vulkan.convert_channels_to_width_packed{125, 500, 1}                    1273428
vulkan.mm                {125, 125, 1}                   77055680
vulkan.mul_scalar        {500, 500, 1}                    1111708
vulkan.mul_scalar        {500, 500, 1}                    1111032
vulkan.add               {500, 500, 1}                    4236024
vulkan.image_to_nchw     {500, 500, 1}                    1429480
...
addmm_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1                       51.1 ms         76.0 ms            5

create_linear_context_benchmark

Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4332432
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1235884
...
create_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1       9.74 ms         30.6 ms            5

run_linear_context_benchmark

Kernel Name              Workgroup Size             Duration (ns)
===========              ==============               ===========
vulkan.nchw_to_image     {500, 500, 1}                    4289740
vulkan.convert_channels_to_height_packed{500, 125, 1}                    1227928
...
run_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1          50.4 ms         86.0 ms            5

full result in P887658084

Reviewed By: liuk22

Differential Revision: D51506293

Summary: Due to recent changes in D51421256 and D51379737, - shaders of `mm`, `addmm`, `bmm`, `baddbmm` are reduced into just `mm`, - height and width packing logic is applied to linear operations so the current perf testings of `addmm` and `create_linear_context` and `run_linear_context` are no longer valid (0 latency will be printed, see test plan). Specifically, the original test extracts latency of `vulkan.addmm` which doesn't exist any more. Instead the current implementation of `addmm` invokes ``` vulkan.convert_channels_to_height_packed vulkan.convert_channels_to_width_packed vulkan.mm vulkan.mul_scalar vulkan.add ``` To deal with this - for `addmm` and `run_linear_context`, we apply a new function `extractTotalShaderResultsAndSetState` which aggregates latency of all invoded shaders except `nchw_to_image` and `image_to_nchw`; - for `create_linear_context`, besides `nchw_to_image` and `image_to_nchw`, we also aggregate `vulkan.convert_channels_to_height_packed` Test Plan: - build binary, at `fbsource` ``` buck2 build -c ndk.debug_info_level=0 -c ndk.static_linking=true -c pt.enable_qpl=0 -c pt.vulkan_use_gpu_diagnostics=1 --target-platforms=ovr_config//platform/android:arm32-fbsource //xplat/caffe2:pt_vulkan_mm_perf_test_binAndroid --show-output -c pt.vulkan_full_precision=1 ``` - test on android device ``` adb push buck-out/v2/gen/fbsource/f1f3f9bed27e143c/xplat/caffe2/__pt_vulkan_mm_perf_test_binAndroid__/pt_vulkan_mm_perf_test_binAndroid /data/local/tmp adb shell /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid ``` ## Before addmm_benchmark ``` (base) luwei@luwei-mbp ~ % adb shell /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid 2023-11-16T06:48:18+00:00 Running /data/local/tmp/pt_vulkan_mm_perf_test_binAndroid Run on (4 X 1708.8 MHz CPU s) ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. ... Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4334408 vulkan.nchw_to_image {500, 500, 1} 4327648 vulkan.nchw_to_image {500, 500, 1} 4322760 vulkan.convert_channels_to_height_packed{500, 125, 1} 1233960 vulkan.convert_channels_to_width_packed{125, 500, 1} 1286896 vulkan.mm {125, 125, 1} 76186084 vulkan.mul_scalar {500, 500, 1} 1132924 vulkan.mul_scalar {500, 500, 1} 1128556 vulkan.add {500, 500, 1} 4285788 vulkan.image_to_nchw {500, 500, 1} 1421576 ... addmm_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 0.000 ms 77.2 ms 5 ``` create_linear_context_benchmark ``` Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4336696 vulkan.convert_channels_to_height_packed{500, 125, 1} 1229384 ... create_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 8.57 ms 32.9 ms 5 ``` run_linear_context_benchmark ``` Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4305548 vulkan.convert_channels_to_height_packed{500, 125, 1} 1196104 ... run_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 0.000 ms 86.2 ms 5 ``` ## After addmm_benchmark ``` Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4332016 vulkan.nchw_to_image {500, 500, 1} 4321356 vulkan.nchw_to_image {500, 500, 1} 4314908 vulkan.convert_channels_to_height_packed{500, 125, 1} 1195896 vulkan.convert_channels_to_width_packed{125, 500, 1} 1273428 vulkan.mm {125, 125, 1} 77055680 vulkan.mul_scalar {500, 500, 1} 1111708 vulkan.mul_scalar {500, 500, 1} 1111032 vulkan.add {500, 500, 1} 4236024 vulkan.image_to_nchw {500, 500, 1} 1429480 ... addmm_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 51.1 ms 76.0 ms 5 ``` create_linear_context_benchmark ``` Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4332432 vulkan.convert_channels_to_height_packed{500, 125, 1} 1235884 ... create_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 9.74 ms 30.6 ms 5 ``` run_linear_context_benchmark ``` Kernel Name Workgroup Size Duration (ns) =========== ============== =========== vulkan.nchw_to_image {500, 500, 1} 4289740 vulkan.convert_channels_to_height_packed{500, 125, 1} 1227928 ... run_linear_context_benchmark/N:500/M:500/P:500/iterations:5/manual_time/threads:1 50.4 ms 86.0 ms 5 ``` full result in P887658084 Reviewed By: liuk22 Differential Revision: D51506293

pytorch-bot · 2023-11-27T20:42:31Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114624

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a41597e with merge base e0d2a24 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-11-27T20:42:36Z

This pull request was exported from Phabricator. Differential Revision: D51506293

facebook-github-bot · 2023-11-28T22:24:06Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2023-11-28T22:27:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR module: vulkan topic: not user facing topic category labels Nov 27, 2023

facebook-github-bot added the fb-exported label Nov 27, 2023

yipjustin approved these changes Nov 28, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 28, 2023

pytorchmergebot added the merging label Nov 28, 2023

pytorchmergebot added Merged and removed merging labels Nov 28, 2023

pytorchmergebot closed this in a26d747 Nov 28, 2023

facebook-github-bot deleted the export-D51506293 branch December 2, 2023 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PyTorch][Vulkan] Fix matrix multiplication performance test binary #114624

[PyTorch][Vulkan] Fix matrix multiplication performance test binary #114624

Uh oh!

copyrightly commented Nov 27, 2023

Uh oh!

pytorch-bot bot commented Nov 27, 2023 •

edited

Loading

Uh oh!

facebook-github-bot commented Nov 27, 2023

Uh oh!

facebook-github-bot commented Nov 28, 2023

Uh oh!

pytorchmergebot commented Nov 28, 2023

Uh oh!

Uh oh!

[PyTorch][Vulkan] Fix matrix multiplication performance test binary #114624

[PyTorch][Vulkan] Fix matrix multiplication performance test binary #114624

Uh oh!

Conversation

copyrightly commented Nov 27, 2023

Before

After

Uh oh!

pytorch-bot bot commented Nov 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114624

✅ No Failures

Uh oh!

facebook-github-bot commented Nov 27, 2023

Uh oh!

facebook-github-bot commented Nov 28, 2023

Uh oh!

pytorchmergebot commented Nov 28, 2023

Merge started

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 27, 2023 •

edited

Loading