[interformer] batch pointwise op + unbind stack pass in post grad #126959

mengluy0125 · 2024-05-23T05:19:41Z

Summary: Tested on H100 with single GPU, and the bs is set to 64.

Test Plan:

local script

buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64

baseline: P1370993922

Metric	Value
Latency	120.84 ms
Model size	5.93 G bytes
Flops/example	62.22 GB
TFLOPS	32.95
MFU	4.12%
Activation/example	128.17 MB

proposal: P1371676068

config

torch._inductor.config.pre_grad_fusion_options = {}
torch._inductor.config.post_grad_fusion_options = {
        "batch_aten_mul": {"min_fuse_set_size": 50},
        "batch_aten_sigmoid": {"min_fuse_set_size": 50},
        "batch_aten_relu": {"min_fuse_set_size": 50},
        "batch_linear_post_grad": {"min_fuse_set_size": 50},
        "unbind_stack_aten_pass": {},
}

Metric	Value
Latency	117.30 ms
Model size	5.93 G bytes
Flops/example	62.65 GB
TFLOPS	34.18
MFU	4.27%
Activation/example	163.12 MB

Differential Revision: D57595173

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

pytorch-bot · 2024-05-23T05:19:44Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126959

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Upgrade MacOS runner to 14

✅ You can merge normally! (13 Unrelated Failures)

As of commit c0e0b38 with merge base 12d6446 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-194002-utc.log'
inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-194021-utc.log'
inductor / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-194156-utc.log'
inductor / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-194136-utc.log'
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-194119-utc.log'
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-193829-utc.log'
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-193905-utc.log'
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (matched linux rule in flaky-rules.json)
No space left on device : '/home/ec2-user/actions-runner/_diag/Worker_20240530-193847-utc.log'

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Lint / lintrunner-noclang / linux-job (gh) (trunk failure)
>>> Lint for test/test_custom_ops.py:

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#127438)
sebotnet33ts_256
inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (inductor_torchbench_cpu_smoketest_perf, 1, 1, linux.24xl.spr-metal, unstable) (gh) (#126993)
Process completed with exit code 1.
pull / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) ()
inductor/test_efficient_conv_bn_eval.py::EfficientConvBNEvalCudaTests::test_basic_cuda
trunk / macos-13-py3-arm64 / test (default, 3, 3, macos-m1-stable, unstable) (gh) ()
functorch/test_ops.py::TestOperatorsCPU::test_grad_svd_lowrank_cpu_float32

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-05-23T05:19:53Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | Differential Revision: D57595173

facebook-github-bot · 2024-05-23T17:23:47Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | Differential Revision: D57595173

facebook-github-bot · 2024-05-23T17:24:23Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…26959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-28T17:59:58Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-28T18:01:15Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-29T19:03:51Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-29T20:45:36Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-29T21:41:57Z

This pull request was exported from Phabricator. Differential Revision: D57595173

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_10_58_35_trace.json.gz&bucket=pyper_traces | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 trace: https://our.intern.facebook.com/intern/perfdoctor/trace_view?filepath=tree/traces/test/interformer.May_22_22_08_09_trace.json.gz&bucket=pyper_traces config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` Counter({'pattern_matcher_nodes': 4677, 'pattern_matcher_count': 3621, 'extern_calls': 1096, 'batch_linear_post_grad': 3, 'batch_aten_mul': 2}) | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | ### experiments after remove split 1. Patch D57739439 2. Detailed experiments can be found in P1376777898 ### Observations: - Batch fusion will introduce stack/cat nodes, which is costly. It only helps when we batch "enough" number of ops; otherwise regression can be seen (e.g., we use the default fusion size and the latency is increased P1370978074) - unbind stack pass are not always taking into effect only when we have nice structure and order, e.g., the example from test_group_batch_fusion, we can delete stack nodes introduced from batch_relu and batch_sigmoid. - Even remove the splits node cannot remove the "batch linear" nodes since the addmm = mm + add, where add leverages the broadcast rule. - Tested with enabling add broadcast and then stack the add ops to cancel the stack, it also not very helpful 1) broadcast is not for free, 2) linear will always introduce two stack nodes (stack input + stack other), which cannot be eliminated. Reviewed By: jackiexu1992 Differential Revision: D57595173

facebook-github-bot · 2024-05-30T17:23:08Z

This pull request was exported from Phabricator. Differential Revision: D57595173

facebook-github-bot · 2024-05-31T03:53:01Z

@pytorchbot merge -f 'Landed internally'

(Initiating merge automatically since Phabricator Diff has merged, using force because this PR might not pass merge_rules.json but landed internally)

pytorchmergebot · 2024-05-31T03:54:34Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…torch#126959) Summary: Tested on H100 with single GPU, and the bs is set to 64. Test Plan: # local script ``` buck2 run mode/opt scripts/jackiexu0313/pt2:uniarch_perf_benchmark -- single-module-benchmark --provider interformer --enable_pt2 True --batch_size 64 ``` baseline: P1370993922 | Metric | Value | |:-------------------|:-------------| | Latency | 120.84 ms | | Model size | 5.93 G bytes | | Flops/example | 62.22 GB | | TFLOPS | 32.95 | | MFU | 4.12% | | Activation/example | 128.17 MB | proposal: P1371676068 config ``` torch._inductor.config.pre_grad_fusion_options = {} torch._inductor.config.post_grad_fusion_options = { "batch_aten_mul": {"min_fuse_set_size": 50}, "batch_aten_sigmoid": {"min_fuse_set_size": 50}, "batch_aten_relu": {"min_fuse_set_size": 50}, "batch_linear_post_grad": {"min_fuse_set_size": 50}, "unbind_stack_aten_pass": {}, } ``` | Metric | Value | |:-------------------|:-------------| | Latency | 117.30 ms | | Model size | 5.93 G bytes | | Flops/example | 62.65 GB | | TFLOPS | 34.18 | | MFU | 4.27% | | Activation/example | 163.12 MB | Differential Revision: D57595173 Pull Request resolved: pytorch#126959 Approved by: https://github.com/jackiexu1992

pytorch-bot bot added ciflow/inductor module: inductor labels May 23, 2024

facebook-github-bot added the fb-exported label May 23, 2024

mengluy0125 force-pushed the export-D57595173 branch from 148f87e to 67a8133 Compare May 23, 2024 17:23

mengluy0125 force-pushed the export-D57595173 branch from 67a8133 to 36959a5 Compare May 23, 2024 17:24

jackiexu1992 approved these changes May 23, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 23, 2024

mengluy0125 force-pushed the export-D57595173 branch from 36959a5 to 8060625 Compare May 28, 2024 17:59

mengluy0125 force-pushed the export-D57595173 branch from 8060625 to 812ee22 Compare May 28, 2024 18:01

mengluy0125 force-pushed the export-D57595173 branch from 812ee22 to cb45e61 Compare May 29, 2024 19:03

mengluy0125 force-pushed the export-D57595173 branch from cb45e61 to 323a974 Compare May 29, 2024 20:45

mengluy0125 force-pushed the export-D57595173 branch from 323a974 to 51f66a5 Compare May 29, 2024 21:41

mengluy0125 force-pushed the export-D57595173 branch from 51f66a5 to c0e0b38 Compare May 30, 2024 17:22

pytorchmergebot added the merging label May 31, 2024

pytorchmergebot closed this in f264745 May 31, 2024

pytorchmergebot added Merged and removed merging labels May 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[interformer] batch pointwise op + unbind stack pass in post grad #126959

[interformer] batch pointwise op + unbind stack pass in post grad #126959

mengluy0125 commented May 23, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented May 23, 2024 •

edited

Loading

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 28, 2024

facebook-github-bot commented May 28, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 30, 2024

facebook-github-bot commented May 31, 2024

pytorchmergebot commented May 31, 2024

[interformer] batch pointwise op + unbind stack pass in post grad #126959

[interformer] batch pointwise op + unbind stack pass in post grad #126959

Conversation

mengluy0125 commented May 23, 2024 • edited by pytorch-bot bot Loading

local script

pytorch-bot bot commented May 23, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126959

❗ 1 Active SEVs

✅ You can merge normally! (13 Unrelated Failures)

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 23, 2024

facebook-github-bot commented May 28, 2024

facebook-github-bot commented May 28, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 29, 2024

facebook-github-bot commented May 30, 2024

facebook-github-bot commented May 31, 2024

pytorchmergebot commented May 31, 2024

Merge started

mengluy0125 commented May 23, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented May 23, 2024 •

edited

Loading