dnn: add the CANN backend #22634

fengyuentau · 2022-10-13T09:36:02Z

merge with adding a wiki page: https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc

Benchmark

Environment (provided by Ascend):

CPU: 20-core aarch64, freq unknown
NPU: Ascend 310

Time is in millisecond. The time of first run is excluded.

Model	CPU backend	CANN backend (this PR)	Native CANN
PP-ResNet50	69.74	3.29	2.51
MobileNetV1	6.60	1.21	0.85
YOLOX	265.90	12.80	9.11
AlexNet	13.19	3.82	3.36

Notes:

PP-ResNet50, MobileNetV1 and YOLOX are from https://github.com/opencv/opencv_zoo.
Native CANN stands for loading and inferring a ATC-converted OM model with CANN interfaces. ATC is the model conversion from ONNX/TF/CAFFE to OM tool provided by CANN. Native CANN only measures the time of aclmdlExecute.
CANN backend (this PR) stands for loading ONNX/TF/CAFFE models with opencv parsers and inferring the OM model built by CANN backend. CANN backend (this PR) measures the time of net.forward(), including aclmdlExecute and some other overheads. Models built by CANN backend and converted by ATC have the same time of aclmdlExecute.
Differences between the implementations of this PR and the previous one:
- This PR: use aclgrphBuildModel to fully optimize the graph and use aclmdlExecute to forward the graph directly.
- Previous: use aclopCompileAndExecute to call and run operator on the fly.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

modules/dnn/CMakeLists.txt

modules/dnn/test/test_onnx_importer.cpp

modules/dnn/src/op_cann.cpp

alalek · 2022-10-13T10:24:02Z

modules/dnn/CMakeLists.txt

+if(HAVE_CANN)
+  list(APPEND include_dirs ${CANN_INCLUDE_DIRS})
+  list(APPEND libs -Wl,--whole-archive ${CANN_LIBRARIES} -Wl,--no-whole-archive)
+endif()


Need to reuse ocv_add_external_target() approach for 3rdparty libraries.

How about keep it as-is since other 3rdparty libraries are still using the old approach? We can do this kind of change for all other 3rdparty libraries in a separate pr.

Do we still need this? (as there is ocv.3rdparty.cann usage below)

Yes, we do. ocv.3rdparty.cann is used to link for opencv_test_dnn only for now.

modules/dnn/src/onnx/onnx_importer.cpp

modules/dnn/src/op_cann.cpp

modules/dnn/test/test_onnx_importer.cpp

alalek · 2022-12-16T09:24:27Z

modules/dnn/CMakeLists.txt

+if(HAVE_CANN)
+  list(APPEND include_dirs ${CANN_INCLUDE_DIRS})
+  list(APPEND libs -Wl,--whole-archive ${CANN_LIBRARIES} -Wl,--no-whole-archive)
+endif()


Do we still need this? (as there is ocv.3rdparty.cann usage below)

modules/dnn/src/layer.cpp

…backend

HaoWangDL · 2023-03-01T07:02:11Z

can support siamrpn use cann backend?
thank you

fengyuentau · 2023-03-01T07:05:26Z

can support siamrpn use cann backend?

Working on the support of DaSiamRPN with CANN backend.

HaoWangDL · 2023-03-01T07:17:21Z

thank u. I will have a try.

HaoWangDL · 2023-03-04T06:39:52Z

can support siamrpn use cann backend?

Working on the support of DaSiamRPN with CANN backend.

I try yolox and classify in model zoo with cann backend, it works.
But i try dasiamrpn(model zoo) with cann backend, encounterd some errors.

[ERROR:0@6.253] global net_cann.cpp:286 compileCannGraph CANN graph check failed, ret = 4294967295 [ERROR:0@8.231] global net_cann.cpp:286 compileCannGraph CANN graph check failed, ret = 4294967295 [ERROR:0@9.299] global op_cann.cpp:258 loadToDevice CANN check failed, ret = 100000 FATAL: C++ exception: OpenCV(4.7.0) /home/ubuntu/Desktop/opencv_git/opencv/modules/dnn/src/op_cann.cpp:258: error: (-2:Unspecified error) CANN check failed in function 'loadToDevice'

can help me? thank u.

fengyuentau · 2023-03-08T08:43:21Z

I am afraid that DaSiamRPN on CANN backend will not be fixed in recent future due to some complicated issues. Actually we are planning to add some other better object tracking models into the zoo to replace DaSiamRPN. They are much faster and more lightweight without sacrificing too much accuracy. I will try to make those new models working on CANN backend instead.

Stay tune for updates!

HaoWangDL · 2023-03-08T08:57:09Z

Looking forward to your update. Thanks.

…

---Original--- From: "Yuantao ***@***.***> Date: Wed, Mar 8, 2023 16:43 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [opencv/opencv] dnn: add the CANN backend (PR #22634) I am afraid that DaSiamRPN on CANN backend will not be fixed in recent future due to some complicated issues. Actually we are planning to add some other better object tracking models into the zoo to replace DaSiamRPN. They are much faster and more lightweight without sacrificing too much accuracy. I will try to make those new models working on CANN backend instead. Stay tune for updates! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

* cann backend impl v1 * cann backend impl v2: use opencv parsers to build models for cann * adjust fc according to the new transA and transB * put cann net in cann backend node and reuse forwardLayer * use fork() to create a child process and compile cann model * remove legacy code * remove debug code * fall bcak to CPU backend if there is one layer not supoorted by CANN backend * fix netInput forward

CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI. Opencv DNN has already suppoted CANN backend [#22634](opencv/opencv#22634). There are more and more users using [Ascend NPU](https://www.hiascend.com/) and programming with CANN, and the number is still growing rapidly. AI training and inference are inseparable from data preprocessing. When users use OpenCV to work with CANN backend, data preprocessing can only run on CPUs, resulting in inefficiency. The purpose of this commit is to enable OpenCV operators on CANN backend. Note that these operators only support Ascend 310 now, other NPUs are theoretically supported, but not tested. The usage of CANN backend is consistent, Please refer to OpenCV DNN: [CANN backend manual] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#file-a_ocv_cann-md): 1. [Install dependencies] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-dependencies) 2. [Install CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-cann) 3. [Compile OpenCV with CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#build-opencv-with-cann) The CANN backend is used in a similar way to CUDA: | Object | CANN | CUDA | | --------- | --------- | -------- | | Namespace | cv::cann | cv::cuda | | Matrix | AclMat | GpuMat | | Stream | AclStream | Stream | | Event | AclEvent | Event | The current commit provides CANN backend operator support framework, In order to make code viewing easy, only 7 basic interfaces are implemented, all of the following operators are tested and compared result with CPU backend: - [x] Add - [x] subtract - [x] multiply - [x] divide - [x] bitwise_and - [x] bitwise_or - [x] bitwise_xor More operators will continue implement in new independent commits. Performance test shows these operators efficiency improved by 2 times. Co-authored-by: CaoMengqing <cmq0113@163.com>

CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI. Opencv DNN has already suppoted CANN backend [#22634](opencv/opencv#22634). There are more and more users using [Ascend NPU](https://www.hiascend.com/) and programming with CANN, and the number is still growing rapidly. AI training and inference are inseparable from data preprocessing. When users use OpenCV to work with CANN backend, data preprocessing can only run on CPUs, resulting in inefficiency. The purpose of this commit is to enable OpenCV operators on CANN backend. Note that these operators only support Ascend 310 now, other NPUs are theoretically supported, but not tested. The usage of CANN backend is consistent, Please refer to OpenCV DNN: [CANN backend manual] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#file-a_ocv_cann-md): 1. [Install dependencies] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-dependencies) 2. [Install CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-cann) 3. [Compile OpenCV with CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#build-opencv-with-cann) The CANN backend is used in a similar way to CUDA: | Object | CANN | CUDA | | --------- | --------- | -------- | | Namespace | cv::cann | cv::cuda | | Matrix | AclMat | GpuMat | | Stream | AclStream | Stream | | Event | AclEvent | Event | The current commit provides CANN backend operator support framework, In order to make code viewing easy, only 7 basic interfaces are implemented, all of the following operators are tested and compared result with CPU backend: - [x] Add - [x] subtract - [x] multiply - [x] divide - [x] bitwise_and - [x] bitwise_or - [x] bitwise_xor More operators will continue implement in new independent commits. Co-authored-by: CaoMengqing <cmq0113@163.com>

CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI. Opencv DNN has already suppoted CANN backend [#22634](opencv/opencv#22634). There are more and more users using [Ascend NPU](https://www.hiascend.com/) and programming with CANN, and the number is still growing rapidly. AI training and inference are inseparable from data preprocessing. When users use OpenCV to work with CANN backend, data preprocessing can only run on CPUs, resulting in inefficiency. The purpose of this commit is to enable OpenCV operators on CANN backend. The usage of CANN backend is consistent, Please refer to OpenCV DNN: [CANN backend manual] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#file-a_ocv_cann-md): 1. [Install dependencies] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-dependencies) 2. [Install CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-cann) 3. [Compile OpenCV with CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#build-opencv-with-cann) The CANN backend is used in a similar way to CUDA: | Object | CANN | CUDA | | --------- | --------- | -------- | | Namespace | cv::cann | cv::cuda | | Matrix | AclMat | GpuMat | | Stream | AclStream | Stream | | Event | AclEvent | Event | The current commit provides CANN backend operator support framework, In order to make code viewing easy, only 7 basic interfaces are implemented, all of the following operators are tested and compared result with CPU backend: - [x] Add - [x] subtract - [x] multiply - [x] divide - [x] bitwise_and - [x] bitwise_or - [x] bitwise_xor More operators will continue implement in new independent commits. Co-authored-by: CaoMengqing <cmq0113@163.com>

CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI. Opencv DNN has already suppoted CANN backend [#22634](opencv/opencv#22634). There are more and more users using [Ascend NPU](https://www.hiascend.com/) and programming with CANN, and the number is still growing rapidly. AI training and inference are inseparable from data preprocessing. When users use OpenCV to work with CANN backend, data preprocessing can only run on CPUs, resulting in inefficiency. The purpose of this commit is to enable OpenCV operators on CANN backend. The usage of CANN backend is consistent, Please refer to OpenCV DNN: [CANN backend manual] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#file-a_ocv_cann-md): 1. [Install dependencies] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-dependencies) 2. [Install CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-cann) 3. [Compile OpenCV with CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#build-opencv-with-cann) The CANN backend is used in a similar way to CUDA: | Object | CANN | CUDA | | --------- | --------- | -------- | | Namespace | cv::cann | cv::cuda | | Matrix | AclMat | GpuMat | | Stream | AclStream | Stream | | Event | AclEvent | Event | The current commit provides CANN backend operator support framework, In order to make code viewing easy, only a few basic interfaces are implemented, all of the following operators are tested and compared result with CPU backend. More operators will continue implement in new independent commits. Co-authored-by: CaoMengqing <cmq0113@163.com>

CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI. Opencv DNN has already suppoted CANN backend [#22634](opencv/opencv#22634). There are more and more users using [Ascend NPU](https://www.hiascend.com/) and programming with CANN, and the number is still growing rapidly. AI training and inference are inseparable from data preprocessing. When users use OpenCV to work with CANN backend, data preprocessing can only run on CPUs, resulting in inefficiency. The purpose of this commit is to enable OpenCV operators on CANN backend. The usage of CANN backend is consistent, Please refer to OpenCV DNN: [CANN backend manual] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#file-a_ocv_cann-md): 1. [Install dependencies] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-dependencies) 2. [Install CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#install-cann) 3. [Compile OpenCV with CANN] (https://gist.github.com/fengyuentau/083f7f339592545c1f1d2c1fde6a53dc#build-opencv-with-cann) The CANN backend is used in a similar way to CUDA: | Object | CANN | CUDA | | --------- | ------------ | -------- | | Namespace | cv::cann | cv::cuda | | Matrix | AscendMat | GpuMat | | Stream | AscendStream | Stream | | Event | AscendEvent | Event | The current commit provides CANN backend operator support framework, In order to make code viewing easy, only a few basic interfaces are implemented, all of the following operators are tested and compared result with CPU backend. More operators will continue implement in new independent commits. Co-authored-by: CaoMengqing <cmq0113@163.com>

fengyuentau mentioned this pull request Oct 13, 2022

Support Huawei Ascend as inference backend and Huawei NPU as inference target in OpenCV DNN #22258

Closed

6 tasks

fengyuentau commented Oct 13, 2022

View reviewed changes

modules/dnn/CMakeLists.txt Outdated Show resolved Hide resolved

fengyuentau commented Oct 13, 2022

View reviewed changes

modules/dnn/test/test_onnx_importer.cpp Outdated Show resolved Hide resolved

fengyuentau commented Oct 13, 2022

View reviewed changes

modules/dnn/test/test_onnx_importer.cpp Outdated Show resolved Hide resolved

fengyuentau commented Oct 13, 2022

View reviewed changes

modules/dnn/src/op_cann.cpp Outdated Show resolved Hide resolved

asmorkalov added feature category: dnn labels Oct 13, 2022

alalek reviewed Oct 13, 2022

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

alalek reviewed Oct 13, 2022

View reviewed changes

modules/dnn/src/op_cann.cpp Outdated Show resolved Hide resolved

fengyuentau force-pushed the cann_backend_221010 branch from 212f5ea to 1a2b855 Compare November 1, 2022 02:39

fengyuentau force-pushed the cann_backend_221010 branch from 20cdd2b to 16946d9 Compare November 16, 2022 02:39

fengyuentau added this to the 4.7.0 milestone Nov 25, 2022

fengyuentau force-pushed the cann_backend_221010 branch 2 times, most recently from 17eb8bd to 89a8e93 Compare November 25, 2022 09:11

fengyuentau added 2 commits December 14, 2022 16:21

cann backend impl v1

ffc8ef0

cann backend impl v2: use opencv parsers to build models for cann

33ea98b

fengyuentau force-pushed the cann_backend_221010 branch from 89a8e93 to 33ea98b Compare December 14, 2022 09:06

fengyuentau added 4 commits December 14, 2022 17:43

adjust fc according to the new transA and transB

00ed7bf

put cann net in cann backend node and reuse forwardLayer

22cdc16

use fork() to create a child process and compile cann model

3863169

remove legacy code

5a1458d

fengyuentau requested a review from vpisarev December 16, 2022 08:03

vpisarev approved these changes Dec 16, 2022

View reviewed changes

remove debug code

5f8befe

alalek reviewed Dec 16, 2022

View reviewed changes

fengyuentau added 2 commits December 16, 2022 20:27

fall bcak to CPU backend if there is one layer not supoorted by CANN …

fb3056f

…backend

fix netInput forward

a95b6e4

vpisarev merged commit a2b3acf into opencv:4.x Dec 21, 2022

alalek mentioned this pull request Jan 8, 2023

(5.x) Merge 4.x #23113

Merged

fengyuentau mentioned this pull request Jan 15, 2023

Update benchmark results on the CANN backend on Ascend 310 opencv/opencv_zoo#117

Closed

hipudding mentioned this pull request Aug 30, 2023

Add operators support for Ascend NPU (CANN backend) opencv/opencv_contrib#3552

Merged

23 tasks

fengyuentau added the category:dnn_cann CANN backend related issues in DNN module label Sep 13, 2023

fengyuentau deleted the cann_backend_221010 branch June 28, 2024 09:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dnn: add the CANN backend #22634

dnn: add the CANN backend #22634

fengyuentau commented Oct 13, 2022 •

edited

Loading

alalek Oct 13, 2022

fengyuentau Oct 18, 2022 •

edited

Loading

alalek Dec 16, 2022

fengyuentau Dec 16, 2022

alalek Dec 16, 2022

HaoWangDL commented Mar 1, 2023

fengyuentau commented Mar 1, 2023

HaoWangDL commented Mar 1, 2023

HaoWangDL commented Mar 4, 2023

fengyuentau commented Mar 8, 2023

HaoWangDL commented Mar 8, 2023 via email

dnn: add the CANN backend #22634

dnn: add the CANN backend #22634

Conversation

fengyuentau commented Oct 13, 2022 • edited Loading

Benchmark

Pull Request Readiness Checklist

alalek Oct 13, 2022

Choose a reason for hiding this comment

fengyuentau Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

alalek Dec 16, 2022

Choose a reason for hiding this comment

fengyuentau Dec 16, 2022

Choose a reason for hiding this comment

alalek Dec 16, 2022

Choose a reason for hiding this comment

HaoWangDL commented Mar 1, 2023

fengyuentau commented Mar 1, 2023

HaoWangDL commented Mar 1, 2023

HaoWangDL commented Mar 4, 2023

fengyuentau commented Mar 8, 2023

HaoWangDL commented Mar 8, 2023 via email

fengyuentau commented Oct 13, 2022 •

edited

Loading

fengyuentau Oct 18, 2022 •

edited

Loading