[QNN] MatMulAddFusion and Reshape Related Fusion#22494
Merged
Conversation
Lafi7e
force-pushed
the
weicwang/matmul_add_fusion
branch
from
7d3d515 to
0a05430
Compare
snnn
previously approved these changes
Oct 21, 2024
skottmckay
reviewed
Oct 22, 2024
Lafi7e
force-pushed
the
weicwang/matmul_add_fusion
branch
from
a8388b7 to
ca59611
Compare
Lafi7e
changed the title
Contributor
|
@centwang Thank you for the PR. It looks like many unit tests and pipelines are still not passing. Could you please address those issues first? |
skottmckay
reviewed
Nov 8, 2024
Lafi7e
force-pushed
the
weicwang/matmul_add_fusion
branch
from
ca59611 to
47d4755
Compare
skottmckay
reviewed
Nov 28, 2024
onnxruntime/core/providers/qnn/builder/qnn_node_group/reshape_gemm_fusion.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/qnn/builder/qnn_node_group/reshape_gemm_fusion.cc
Outdated
Show resolved
Hide resolved
skottmckay
previously approved these changes
Jan 17, 2025
onnxruntime/core/providers/qnn/builder/qnn_node_group/reshape_gemm_fusion.cc
Outdated
Show resolved
Hide resolved
adrianlizarraga
previously approved these changes
Feb 12, 2025
adrianlizarraga
requested review from
HectorSVC
and removed request for
cloudhan
Contributor
|
@HectorSVC could you please take a look at this PR? |
adrianlizarraga
approved these changes
Feb 14, 2025
HectorSVC
approved these changes
Feb 14, 2025
Contributor
|
Hi @skottmckay, I think there are some unresolved comments. Would you be able to take another look? |
guschmue
pushed a commit
that referenced
this pull request
Mar 6, 2025
QNN EP relies on Gemm Op to use FullyConnected QNN Op to run the model, which is much faster than MatMul+Add. This PR fuses MatMul+Add when MatMul's 2nd input is 2D initializer, no matter the rank of the 1st input. If the 1st input is not 2D tensor, Reshape nodes will be added. On QNN EP, the memory allocation is for each activation tensor, so Reshape/Squeeze/Unsqueeze is not no-op. This PR also add some fusion trying to remove redundant reshape nodes. For some QNN AI Hub models on specific device, without removing the Reshape nodes, it cannot finalize the graph when execution, but works well after removing. Run below models with and without the change: swin_tiny: Average inference time cost: 12.8077 ms | Average inference time cost: 23.956 ms swin_base: Average inference time cost: 27.0639 ms | Average inference time cost: 57.6608 ms convnext_tiny: Average inference time cost: 3.42956 ms | Average inference time cost: 16.1848 ms openai_clip_CLIPTextEncoder: Average inference time cost: 5.96104 ms | Average inference time cost: 220.406 ms openai_clip_CLIPImageEncoder: Average inference time cost: 41.8206 ms | Average inference time cost: 919.712 ms NOTE that current change skips the Attention pattern because it not it will cause AttentionFusion to work. Ideally we need to adjust the AttentionFusion to support the Gemm pattern, but it requires big changes. Maybe we can do this in the future, say, when we want to run transformer models on QNN, since we don't have Attention QNN, we still want to fuse MatMul+Add in the Attention pattern to use FullyConnected in QNN side. --------- Co-authored-by: adrianlizarraga <adlizarraga@microsoft.com>
ashrit-ms
pushed a commit
that referenced
this pull request
Mar 17, 2025
QNN EP relies on Gemm Op to use FullyConnected QNN Op to run the model, which is much faster than MatMul+Add. This PR fuses MatMul+Add when MatMul's 2nd input is 2D initializer, no matter the rank of the 1st input. If the 1st input is not 2D tensor, Reshape nodes will be added. On QNN EP, the memory allocation is for each activation tensor, so Reshape/Squeeze/Unsqueeze is not no-op. This PR also add some fusion trying to remove redundant reshape nodes. For some QNN AI Hub models on specific device, without removing the Reshape nodes, it cannot finalize the graph when execution, but works well after removing. Run below models with and without the change: swin_tiny: Average inference time cost: 12.8077 ms | Average inference time cost: 23.956 ms swin_base: Average inference time cost: 27.0639 ms | Average inference time cost: 57.6608 ms convnext_tiny: Average inference time cost: 3.42956 ms | Average inference time cost: 16.1848 ms openai_clip_CLIPTextEncoder: Average inference time cost: 5.96104 ms | Average inference time cost: 220.406 ms openai_clip_CLIPImageEncoder: Average inference time cost: 41.8206 ms | Average inference time cost: 919.712 ms NOTE that current change skips the Attention pattern because it not it will cause AttentionFusion to work. Ideally we need to adjust the AttentionFusion to support the Gemm pattern, but it requires big changes. Maybe we can do this in the future, say, when we want to run transformer models on QNN, since we don't have Attention QNN, we still want to fuse MatMul+Add in the Attention pattern to use FullyConnected in QNN side. --------- Co-authored-by: adrianlizarraga <adlizarraga@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QNN EP relies on Gemm Op to use FullyConnected QNN Op to run the model, which is much faster than MatMul+Add. This PR fuses MatMul+Add when MatMul's 2nd input is 2D initializer, no matter the rank of the 1st input. If the 1st input is not 2D tensor, Reshape nodes will be added.
On QNN EP, the memory allocation is for each activation tensor, so Reshape/Squeeze/Unsqueeze is not no-op. This PR also add some fusion trying to remove redundant reshape nodes. For some QNN AI Hub models on specific device, without removing the Reshape nodes, it cannot finalize the graph when execution, but works well after removing.
Run below models with and without the change:
swin_tiny: Average inference time cost: 12.8077 ms | Average inference time cost: 23.956 ms
swin_base: Average inference time cost: 27.0639 ms | Average inference time cost: 57.6608 ms
convnext_tiny: Average inference time cost: 3.42956 ms | Average inference time cost: 16.1848 ms
openai_clip_CLIPTextEncoder: Average inference time cost: 5.96104 ms | Average inference time cost: 220.406 ms
openai_clip_CLIPImageEncoder: Average inference time cost: 41.8206 ms | Average inference time cost: 919.712 ms
NOTE that current change skips the Attention pattern because it not it will cause AttentionFusion to work. Ideally we need to adjust the AttentionFusion to support the Gemm pattern, but it requires big changes. Maybe we can do this in the future, say, when we want to run transformer models on QNN, since we don't have Attention QNN, we still want to fuse MatMul+Add in the Attention pattern to use FullyConnected in QNN side.