Switch to new OpenVINO API after 2022.1 release

[dkurt]

• Use OpenVINO Runtime API (aka Tensor API or API 2.0) which has been introduces in 2022.1
• Remove InferenceEngine::DataPtr completely
• Internal dynamism feature works only with a new API. Actual for Faster-* family models and with layers such as NonZero, Top-k and others. This feature may improve performance.

• Tested with OpenVINO 2022.1 (CI)
• Tested with OpenVINO 2021.4 (CI)
• Tested with old OpenVINO (for example, 2020.3) - (alalek: older versions are not supported on 4.x branch)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
• The PR is proposed to the proper branch
• There is a reference to the original bug report and related work
• There is accuracy test, performance test and test data in opencv_extra repository, if applicable
  Patch to opencv_extra has the same branch name.
• The feature is well documented and sample code can be built with the project CMake

force_builders=Custom

Xbuild_image:Custom=ubuntu-openvino-2021.4.2:20.04
build_image:Custom=ubuntu-openvino-2022.1.0:20.04

test_modules:Custom=dnn,python2,python3,java,gapi,video

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=ON
test_bigdata:Custom=1
test_filter:Custom=*

allow_multiple_commits=1

[alalek]

@dkurt Thank you for the update!

CI failures are related to OOM issue (32Gb).

./bin/opencv_test_dnn --gtest_filter=*arknet*

consumes 17Gb with OpenVINO 2022.1 and enabled OpenCL.

Could you have a chance to take a look on this?

[dkurt]

There must be a leak. Can reproduce with a single test on both CPU and GPU:

/usr/bin/time --verbose ./opencv_test_dnn --gtest_repeat=1 --gtest_param_filter=NGRAPH/OCL --gtest_filter=Test_Darknet_nets.YOLOv4x_mish/* 

Num repeats	Top resident memory (NGRAPH/OCL)	Top resident memory (NGRAPH/CPU)
1	2.14GB	3.59GB
2	3.21GB	6.15GB
3	4.27GB	8.71GB

Update: found two problems: asynchronous callback and cycled dependency between InfEngineNgraphNode and InfEngineNgraphNet caused by requestedOutputs so InfEngineNgraphNet destructor was not called.

[alalek]

@dkurt Thank you 👍

[dkurt]

Need to fix failed OCL tests with 2021.4, thanks for configuring.

@alalek, problem is reproduced on 4.x branch. Appears on HETERO:GPU,CPU configuration and fixed in 2022.x OpenVINO release. Solved by adding missed layers so no fallback to CPU is needed.

[JulienMaille]

@alalek is it too soon to try compiling with newly released Openvino?