dnn: add attention layer #24476

fengyuentau · 2023-11-01T08:56:09Z

Resolves #24609

Attention operator spec from onnxruntime: https://github.com/microsoft/onnxruntime/blob/v1.16.1/docs/ContribOperators.md#com.microsoft.Attention.

TODO:

benchmark (before this PR vs. with this PR vs. ORT).
Layer fusion: Take care Slice with end=INT64_MAX.
Layer fusion: match more potential attention (VIT) patterns.
- Single-head attention is supported.
Test AttentionSubgraph fusion.
Add acc tests for VIT_B_32 and VitTrack
Add perf tests for VIT_B_32 and VitTrack

Benchmarks

Platform: Macbook Air M1.

Attention Subgraph

Input scale: [1, 197, 768].

	mean (ms)	median (ms)	min (ms)
w/ Attention (this PR)	3.75	3.68	3.22
w/o Attention	9.06	9.01	8.24
ORT (python)	4.32	2.63	2.50

ViTs

All data in millisecond (ms).

ViTs	With Attention	Without Attention	ORT
vit_b_16	302.77	365.35	109.70
vit_b_32	89.92	116.22	30.36
vit_l_16	1593.32	1730.74	419.92
vit_l_32	468.11	577.41	134.12
VitTrack	3.80	3.87	2.25

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

modules/dnn/include/opencv2/dnn/all_layers.hpp

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

fengyuentau · 2023-11-24T04:23:03Z

Benchmark results are added.

dkurt · 2023-11-24T06:27:06Z

PR is good, thanks a lot. The only concern about the potential regression/fallback in backends because of transition from separate ops to a new fused layer.

However, there is no alternatives for this problem for now and I recommend to keep this PR only for default CPU implementation.

Please also take a look at this comment: opencv/opencv_extra#1128 (comment)

dkurt · 2023-11-24T06:28:51Z

modules/dnn/perf/perf_layer.cpp

+            net.setInput(bias, input_names[2]);
+
+            net.setPreferableBackend(backendId);
+            net.setPreferableTarget(targetId);


I will benchmark on CUDA and with OpenVINO

Current numbers a bit confusing. Let's wait for #24476 (comment) because it fails.

input shape: [1, 320, 48]
model from opencv/opencv_extra#1128
CPU: 12th Gen Intel(R) Core(TM) i9-12900

backend 4.x PR

OpenCV, CPU 18.46ms 11.90ms (x0.64)

OpenVINO, CPU 0.25ms 11.91ms

Well, the acc test have a reduced scale so that it does not take a lot time to run. For benchmark, I use [1, 197, 768] (I have two attention.onnx models, one with input [1, 197, 768], the other one [1, 320, 48]).

For OpenVINO, since we do not have backend-specific graph fusion, we can recreate this subgraph in initNgraph.

Updated results. Please ignore the table above:

input: 1x197x768
CPU: 12th Gen Intel(R) Core(TM) i9-12900

backend 4.x PR

OpenCV, CPU 16.93ms 2.37ms (x7.14)

OpenVINO, CPU 1.54ms 2.33ms (x0.66)

So there is a degradation in OpenVINO performance in case of fallback to OpenCV layer.

fengyuentau · 2023-11-24T06:30:26Z

I will try to implement different backends for this layer in another pull request. Just try to reduce the review and merge this ASAP.

modules/dnn/src/layers/attention_layer.cpp

modules/dnn/test/test_onnx_importer.cpp

modules/dnn/perf/perf_layer.cpp

asmorkalov · 2023-12-01T06:13:28Z

@dkurt @WanliZhong Friendly reminder.
@vpisarev Could you join the PR review.

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

dkurt · 2023-12-01T06:41:36Z

modules/dnn/perf/perf_net.cpp

+| vit_b_32 | 89.92          | 116.22            | 30.36  |
+| vit_l_16 | 1593.32        | 1730.74           | 419.92 |
+| vit_l_32 | 468.11         | 577.41            | 134.12 |
+| VitTrack | 3.80           | 3.87              | 2.25   |


Please remove the results from source code - it's enough to add to PR's description

Running this models in perf_net.cpp is taking up too much time for now. How about we comment PERF_TEST_P_(DNNTestNetwork, VIT_B_16) and such for now until we get the close or even better inference speed as ORT? In this case, these perfamce results should be deleted and kept in the first comment of this PR.

You may add a corresponding skip exception:

applyTestTag(CV_TEST_TAG_LONG, CV_TEST_TAG_DEBUG_LONG);

modules/dnn/src/graph_simplifier.cpp

dkurt

There is no issue if we specify 4 inputs for "Slice". So no need to add new logic with optional inputs:

Track #24609 and proposal #24609 (comment)

fengyuentau · 2023-12-01T14:59:43Z

I dont think the proposal is good enough. We actually have other models that have Step to be non-default value. Can we achieve something like a switch per Subgraph that it omits optinal inputs if turned on?

dkurt · 2023-12-04T07:06:21Z

@fengyuentau, got it. So can you please in this PR do a simpler workaround like a parameter with number of Slice inputs? This problem should be solved separately.

class AttentionSubGraph : public Subgraph {
 public:
    AttentionSubGraph(int numSliceInps) {
        std::vector<std::string> inps(1 + numSliceInps, att_add);
        for (int i = 0; i < numSliceInps; ++i)
            inps[i + 1] = addNodeToMatch("");
        slice_v = addNodeToMatch("Slice", inps);
    }

…; add test for attention subgraph fusion

…t is matched; clean comments

asmorkalov · 2023-12-20T11:54:41Z

CUDA:

[ RUN      ] Test_ONNX_nets.ViT_B_32/1, where GetParam() = CUDA/CUDA_FP16
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:76: Failure
Expected: (normL1) <= (l1), actual: 0.00828554 vs 0.004
ViTB_32  |ref| = 2.8940291404724121
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.0407351 vs 0.02
ViTB_32  |ref| = 2.8940291404724121
[  FAILED  ] Test_ONNX_nets.ViT_B_32/1, where GetParam() = CUDA/CUDA_FP16 (1393 ms)

asmorkalov · 2023-12-20T11:55:56Z

@opencv-alalek Please update test data on Buildbot.

fengyuentau · 2023-12-20T11:58:37Z

CUDA:

[ RUN      ] Test_ONNX_nets.ViT_B_32/1, where GetParam() = CUDA/CUDA_FP16
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:76: Failure
Expected: (normL1) <= (l1), actual: 0.00828554 vs 0.004
ViTB_32  |ref| = 2.8940291404724121
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.0407351 vs 0.02
ViTB_32  |ref| = 2.8940291404724121
[  FAILED  ] Test_ONNX_nets.ViT_B_32/1, where GetParam() = CUDA/CUDA_FP16 (1393 ms)

I was also looking into this issue. It is so weird that it only fails on CUDA_FP16 with almost double the threshold value. Could we apply skip tag CV_TEST_TAG_DNN_SKIP_CUDA_FP16 for now?

asmorkalov · 2023-12-20T12:54:42Z

CV_TEST_TAG_DNN_SKIP_CUDA_FP16 - sure.

asmorkalov

👍

fengyuentau added feature category: dnn category: dnn (onnx) ONNX suport issues in DNN module labels Nov 1, 2023

fengyuentau added this to the 4.9.0 milestone Nov 1, 2023

dkurt reviewed Nov 1, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/all_layers.hpp Outdated Show resolved Hide resolved

dkurt reviewed Nov 1, 2023

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Outdated Show resolved Hide resolved

fengyuentau mentioned this pull request Nov 3, 2023

[Feature Request]: attention operator support? openvinotoolkit/openvino#20850

Open

1 task

fengyuentau force-pushed the attention_layer branch 2 times, most recently from 304584b to 5671dec Compare November 8, 2023 03:36

fengyuentau force-pushed the attention_layer branch from 32293fd to 816c331 Compare November 20, 2023 10:22

fengyuentau mentioned this pull request Nov 23, 2023

dnn: add model and data for attention layer implementation opencv/opencv_extra#1128

Merged

fengyuentau marked this pull request as ready for review November 23, 2023 10:34

fengyuentau assigned vpisarev Nov 23, 2023

dkurt reviewed Nov 24, 2023

View reviewed changes

modules/dnn/src/layers/attention_layer.cpp Show resolved Hide resolved

dkurt reviewed Nov 24, 2023

View reviewed changes

modules/dnn/test/test_onnx_importer.cpp Show resolved Hide resolved

dkurt reviewed Nov 24, 2023

View reviewed changes

modules/dnn/perf/perf_layer.cpp Show resolved Hide resolved

fengyuentau force-pushed the attention_layer branch from 669b503 to b3068a3 Compare November 25, 2023 10:34

WanliZhong mentioned this pull request Nov 29, 2023

Make default axis of softmax in onnx "-1" without opset option #24613

Merged

fengyuentau mentioned this pull request Nov 29, 2023

dnn graph simplifier: support optional constant inputs when match #24609

Closed

dkurt reviewed Dec 1, 2023

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Outdated Show resolved Hide resolved

dkurt reviewed Dec 1, 2023

View reviewed changes

modules/dnn/src/graph_simplifier.cpp Outdated Show resolved Hide resolved

dkurt requested changes Dec 1, 2023

View reviewed changes

fengyuentau added 16 commits December 20, 2023 15:27

add test case

3ee63fd

add perf

751cea7

support v_Slice.end=INT64_MAX; support single-head attention subgraph…

a1128aa

…; add test for attention subgraph fusion

add acc tests (commented for now); clear qkv_hidden_sizes everytime i…

92701ea

…t is matched; clean comments

handle optional inputs in graph simplifier

c8832ca

add perf and acc test for vittrack (comment for now)

61965d6

revert graph simplifier changes before rebase

0d36eac

fix graph simplifier

0776fc5

clear perf results

9e617fa

cpu only attention subgraph fusion

5c12f40

use OPENCV_DNN_BACKEND_DEFAULT

5fce823

changes by review

677694a

slice up to 5 inputs

35c3123

add acc and perf tests

a4f4811

empty commit to trigger tests

2db7246

empty commit to trigger tests 1

846237d

fengyuentau force-pushed the attention_layer branch from 1bc226e to 846237d Compare December 20, 2023 07:29

fengyuentau and others added 2 commits December 20, 2023 16:53

try to make ci green by fixing output dimension problem

51e2f25

fix shape; set weight path of vittrack optional

2e5ea89

Skit new test for CUDA FP16 for now.

cb8ac70

asmorkalov approved these changes Dec 20, 2023

View reviewed changes

asmorkalov merged commit 0521a3a into opencv:4.x Dec 20, 2023
25 of 26 checks passed

fengyuentau deleted the attention_layer branch December 21, 2023 09:12

opencv-alalek mentioned this pull request Dec 21, 2023

dnn(test): tune FP16 test tolerance #24737

Merged

asmorkalov mentioned this pull request Jan 19, 2024

5.x merge 4.x #24862

Merged

fengyuentau mentioned this pull request Feb 21, 2024

ONNX conformance test results #21078

Open

48 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dnn: add attention layer #24476

dnn: add attention layer #24476

fengyuentau commented Nov 1, 2023 •

edited

fengyuentau commented Nov 24, 2023

dkurt commented Nov 24, 2023

dkurt Nov 24, 2023

dkurt Nov 24, 2023

fengyuentau Nov 24, 2023

dkurt Dec 1, 2023

fengyuentau commented Nov 24, 2023

asmorkalov commented Dec 1, 2023

dkurt Dec 1, 2023

fengyuentau Dec 7, 2023

dkurt Dec 7, 2023

dkurt left a comment •

edited

fengyuentau commented Dec 1, 2023

dkurt commented Dec 4, 2023

asmorkalov commented Dec 20, 2023

asmorkalov commented Dec 20, 2023

fengyuentau commented Dec 20, 2023

asmorkalov commented Dec 20, 2023

asmorkalov left a comment

backend	4.x	PR
OpenCV, CPU	16.93ms	2.37ms (x7.14)
OpenVINO, CPU	1.54ms	2.33ms (x0.66)

dnn: add attention layer #24476

dnn: add attention layer #24476

Conversation

fengyuentau commented Nov 1, 2023 • edited

Benchmarks

Attention Subgraph

ViTs

Pull Request Readiness Checklist

fengyuentau commented Nov 24, 2023

dkurt commented Nov 24, 2023

dkurt Nov 24, 2023

Choose a reason for hiding this comment

dkurt Nov 24, 2023

Choose a reason for hiding this comment

fengyuentau Nov 24, 2023

Choose a reason for hiding this comment

dkurt Dec 1, 2023

Choose a reason for hiding this comment

fengyuentau commented Nov 24, 2023

asmorkalov commented Dec 1, 2023

dkurt Dec 1, 2023

Choose a reason for hiding this comment

fengyuentau Dec 7, 2023

Choose a reason for hiding this comment

dkurt Dec 7, 2023

Choose a reason for hiding this comment

dkurt left a comment • edited

Choose a reason for hiding this comment

fengyuentau commented Dec 1, 2023

dkurt commented Dec 4, 2023

asmorkalov commented Dec 20, 2023

asmorkalov commented Dec 20, 2023

fengyuentau commented Dec 20, 2023

asmorkalov commented Dec 20, 2023

asmorkalov left a comment

Choose a reason for hiding this comment

fengyuentau commented Nov 1, 2023 •

edited

dkurt left a comment •

edited