Is there any tutorial or example to show how to use Inference Engine models in OpenCV

[Bahramudin]

I want to know if there is a tutorial or a using example to show how to use Inference Engine pre-trained models in OpenCV to detect the objects like face, human, car, etc...

I have already downloaded and installed the Intel® OpenVINO™ toolkit I followed this wiki.

Basically I have two questions
Question 1: I tried to build OpenCV from source with Inference Engine, but the CMake was unable to locate the Inference Engine_DIR, It will be better to also have a tutorial to show how to build, it the wiki above it is not very clear. So I was not able to load the Inference Engine pre-trained model in the OpenCV which was built by me, it has thrwon the exception which say the Inference Engine is not enable.
OK, so I used the OpenCV which came with the OpenVINO toolkit.

Question 2: When I loaded the face-detection-adas-0001 xml and bin using cv::dnn::Net::readxxxxx(xml, bin) it was woking and did not throw any exception, but in the next step I don't know how to pass the frame (cv::Mat) to the Network and get the result. I am looking for an axample to show how to the pre-trained models in OpenCV.

Thanks!!!!

[snosov1]

Hey, @Bahramudin !

I believe, you saw the demos in this repo that show how to do detection of human/face/etc. And your questions are mostly about cv::dnn module, correct?

@dkurt, can you comment on the cv::dnn part?

[dkurt]

@Bahramudin, after cv::dnn::Net::readNet(xml, bin) there is no difference for origin of model.

Take a look on dnn's samples: https://github.com/opencv/opencv/tree/master/samples/dnn.

As for mentioned model, you can use OpenCV's object detection sample:

python /path/to/opencv/samples/dnn/object_detection.py \
    --model /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.bin \
    --config /opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.xml \
    --width 672 --height 384

[dkurt]

Just for example,

import cv2 as cv

net = cv.dnn.readNet('/opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.bin',
                     '/opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.xml')

cap = cv.VideoCapture(0)

while cv.waitKey(1) < 0:
    hasFrame, frame = cap.read()
    if not hasFrame:
        break

    blob = cv.dnn.blobFromImage(frame, size=(672, 384))
    net.setInput(blob)
    out = net.forward()

    for detection in out.reshape(-1, 7):
        confidence = float(detection[2])
        xmin = int(detection[3] * frame.shape[1])
        ymin = int(detection[4] * frame.shape[0])
        xmax = int(detection[5] * frame.shape[1])
        ymax = int(detection[6] * frame.shape[0])

        if confidence > 0.5:
            cv.rectangle(frame, (xmin, ymin), (xmax, ymax), color=(0, 255, 0))

    cv.imshow('OpenVINO face detection', frame)

[Bahramudin]

@snosov1 Sure. This is the OpenCV Dnn::Net Documentation:

   /** @brief Create a network from Intel's Model Optimizer intermediate representation.
     *  @param[in] xml XML configuration file with network's topology.
     *  @param[in] bin Binary file with trained weights.
     *  Networks imported from Intel's Model Optimizer are launched in Intel's Inference Engine
     *  backend.
     */
    CV_WRAP static Net readFromModelOptimizer(const String& xml, const String& bin);

I did something like this:

QString basePath = "C:/Intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/";
QString xml = basePath+"face-detection-adas-0001.xml";
QString bin = basePath+"face-detection-adas-0001.bin";
cv::dnn::Net net  = cv::dnn::Net::readFromModelOptimizer(xml, bin);

cv::VideoCapture capture(0);
cv::Mat frame;
while(capture.read(frame)) {
    // how to pass the frame to the net
    
    // and how to get the result
    cv::waitKey(1);
}

[Bahramudin]

@dkurt Thanks, for the example, I am going to try this now.

[Bahramudin]

@dkurt Can you tell me what is the different betweenreadNet()and readFromModelOptimizer() because I want to use readFromModelOptimizer() instead of readNet(), becuase the documentation says that it is:

Create a network from Intel's Model Optimizer intermediate representation.

And also what is size=(672, 384) ?

What is your comment?

[dkurt]

@Bahramudin, I think the code can tell you more than my words:

https://github.com/opencv/opencv/blob/64813977109a5057600b4da63d5446a97aebbfee/modules/dnn/src/dnn.cpp#L3589-L3635

And referencing it's documentation: https://docs.opencv.org/master/d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422

This function automatically detects an origin framework of trained model and calls an appropriate function such readNetFromCaffe, readNetFromTensorflow, readNetFromTorch or readNetFromDarknet. An order of model and config arguments does not matter.

672x384 is a size of input blob. The network has been trained on images of that size. @snosov1, isn't it?

Actually, object detection models can work well on different input sizes. However for now you can test it only with models in native formats using OpenCV. Inference Engine's feature called shape inference which allows you change input dimensions is not implemented in OpenCV right now.

[Bahramudin]

@dkurt I was able to use the code in your answer and successfully detected the objects, THANKS FOR THAT.
But some model was not able to run, anyway, I want to ask that how to build OpenCV with Inference Engine from source? Because when I run the Inference Engine, this will throw the below error:

OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.0.0-pre) Error: Unspecified error (Build OpenCV with Inference Engine to enable loading models from Model Optimizer.) in cv::dnn::experimental_dnn_v4::Net::readFromModelOptimizer, file d:\github\opencv\opencv\modules\dnn\src\dnn.cpp, line 1995

I tried to build the OpenCV from source with Inference Engine, but CMake was not able to find the Inference Engine DIR,

InferenceEngine_DIR-NOTFOUND

I have done everything as the wiki example.

Thanks!!

[dkurt]

@Bahramudin, could you please specify Inferene Engine version (release of OpenVINO) and output of setupvars.bat?

[Bahramudin]

@dkurt
OpenVINO: 2018 R4

output of setupvars.bat:
Python 3.6.4
ECHO Is off.
PYTHONPATH=C:\Intel\computer_vision_sdk_2018.4.420\python\python3.6;C:/SimpleCV1.3/files/opencv/build/python/2.7/;C:/OpenCV2.3/opencv/build/python/2.7/;
[setupvars.bat] OpenVINO environment initialized

[Bahramudin]

@snosov1 What about the first Question, which asks: How to build OpenCV with Inference Engine

[snosov1]

@Bahramudin To double-check that I understand your question correctly - are you talking about building OpenCV with Inference Engine backend to the cv::dnn module? If yes, then, please, work with @dkurt on that.

If you simply need to work with IE and OpenCV in a single application, then there's really nothing special about that - you just build the libraries independently and link against them in your application.

[Bahramudin]

@snosov1 Yes, you understand correctly. The OpenCV pre-build library which comes with Intel® OpenVINO™ toolkit has already built with IE backend, which can load and able to run the intel_models. But the pre-built library does not contain some other functionality which I need, e.g GStreamer, ... etc. So I want to build the OpenCV by my self with all the needed backends.

Anyway, thanks for your reply!!!

[snosov1]

Ok, got it! But I don't know anybody better than @dkurt who can assist with that =)

[alalek]

GStreamer is available in OpenCV for OpenVINO for Linux builds only.
FFmpeg is available in both Linux / Windows variants (but on Windows you need to download FFmpeg wrapper using provided script ffmpeg-download.ps1).

There are logs of custom OpenCV builds using Inference Engine from OpenVINO installation:
http://pullrequest.opencv.org/buildbot/builders/3_4_openvino-skl-lin64
Additional arguments are here.
Also be careful too avoid conflicts with OpenCV from OpenVINO.

[Bahramudin]

@alalek Thanks!!! Yes, there is GStreamer available for Linux system, but not for Windows, and also the pre-built lib also does not contain opencv_contrib. My first target system is Windows OS. While building OpenCV from source, everything goes well, but only CMake is not able to find the InferenceEngine_DIR-NOTFOUND which when I load the intel_models after build, it throws the exception below:

OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.0.0-pre) Error: Unspecified error (Build OpenCV with Inference Engine to enable loading models from Model Optimizer.) in cv::dnn::experimental_dnn_v4::Net::readFromModelOptimizer, file d:\github\opencv\opencv\modules\dnn\src\dnn.cpp, line 1995

:(

[alalek]

InferenceEngine_DIR-NOTFOUND

You should resolve this problem first.

Specify InferenceEngine_DIR for CMake (like setupvars do).
Debug configuration issues via CMake's "--trace" / "--trace-expand".

[dkurt]

@Bahramudin, what is InferenceEngine_DIR-NOTFOUND?

[Bahramudin]

@dkurt It is CMake error which unable to find IE dir

I have manually given the path to the InferenceEngine, e.g:

1. C:\Intel\computer_vision_sdk_2018.4.420
2. C:\Intel\computer_vision_sdk_2018.4.420\bin
3. C:\Intel\computer_vision_sdk_2018.4.420\inference_engine
4. C:\Intel\computer_vision_sdk_2018.4.420\inference_engine\bin
5. C:\Intel\computer_vision_sdk_2018.4.420\deployment_tools\inference_engine
6. C:\Intel\computer_vision_sdk_2018.4.420\deployment_tools\inference_engine\bin

No one was working.

[asuhov]

@Bahramudin, could you try to set InferenceEngine_DIR=C:\Intel\computer_vision_sdk_2018.4.420\deployment_tools\inference_engine\share?

[Bahramudin]

@snosov1 I have tried the path you mentioned, it seems that cmake was able to find the DIR.

But now it throws this error:

[dkurt]

@Bahramudin, can reproduce your issue using CMake GUI.

[dkurt]

@Bahramudin, as a workaround you may try to comment the following two lines in C:\Intel\computer_vision_sdk_2018.4.420\inference_engine\share\InferenceEngineConfig.cmake:

        add_subdirectory(${IE_SRC_DIR}/extension EXCLUDE_FROM_ALL ie_cpu_extension)
        add_library(IE::ie_cpu_extension ALIAS ie_cpu_extension)

https://github.com/opencv/dldt/blob/55a41d7570f78aaea0d6764d157dd7434730d56f/inference-engine/cmake/share/InferenceEngineConfig.cmake#L167-L168

Please add error messages in text instead of screenshots next time. Otherwise other users can't find this issue if they faced similar problem.

[Bahramudin]

@dkurt Yes, that was solved the problem. By the way, thanks for your suggestion due to add error messages in text instead of screenshots, I will keep it in mind.

One small question that why there are more than one computer_vision_sdk directory in the Inference Engine Toolkit e.g

1. C:\Intel\computer_vision_sdk
2. C:\Intel\computer_vision_sdk_2018.4.420

And also why there are more than one inference_engine inside each computer_vision_sdk directory, e.g

1. C:\Intel\computer_vision_sdk_2018.4.420\inference_engine
2. C:\Intel\computer_vision_sdk_2018.4.420\deployment_tools\inference_engine

This will cnfused people which one to take or use?
If there are all the same, then it will be better clean up the duplicated packages, keep the toolkit clean, fresh, up to date, and small size.

It was just a suggestion, thank you!

[dkurt]

C:\Intel\computer_vision_sdk is a shortcut to the latest installed OpenVINO (C:\Intel\computer_vision_sdk_2018.4.420 in your case).

C:\Intel\computer_vision_sdk_2018.4.420\inference_engine is also shortcut to C:\Intel\computer_vision_sdk_2018.4.420\deployment_tools\inference_engine.

[dkurt]

@nikogamulin, please do not duplicate it everywhere. Let's keep our discussion at forum.

[abhimasand]

@dkurt Hi, I tried using your open-cv code for loading the face detection model, it worked great !! But now I am trying to use the age and gender model and when I'm getting its output, I only get the gender in the form of softmax output, I don't get the age. If you know how to get the age too, please let me know. Thanks in advance.

[dkurt]

@Mashex, forward methods can work with multiple outputs names. Please check it in docs: https://docs.opencv.org/master/db/d30/classcv_1_1dnn_1_1Net.html#adb34d7650e555264c7da3b47d967311b.

[abhimasand]

@dkurt Thank you very much

[snosov1]

Seems like this is a fairly popular topic.

@dkurt do you have any collateral we can reference in the docs? (Since I don't think that this issue is an appropriate place for disseminating information)

[dkurt]

@snosov1, There is a wiki page of how to build OpenCV with IE support https://github.com/opencv/opencv/wiki/Intel%27s-Deep-Learning-Inference-Engine-backend. There is no API difference between OpenCV's dnn and OpenCV's dnn with IE backend. So author should follow samples or tutorials.

[snosov1]

Well, neither samples nor tutorials cover the specific topic in question - "How to use Inference Engine (i.e. Open Model Zoo) models". I understand, that it should more or less follow the general logic, but still we see that there are certain specifics - use this method instead of that method, use forward for networks with multiple outputs, etc. Maybe a tutorial that covers most common pitfalls could be useful.

Anyway, closing this issue for now.

[Jiakui]

any body know how to use the model license-plate-recognition-barrier-0001 in openCV？ there are two inputs, name: "data" and name: "seq_ind" . How to set these two inputs in openCV ? name: "data" can be set by converting image with cv.dnn.blobFromImage, but how to convert name: "seq_ind" for setInput ?

[ada-nai]

any body know how to use the model license-plate-recognition-barrier-0001 in openCV？ there are two inputs, name: "data" and name: "seq_ind" . How to set these two inputs in openCV ? name: "data" can be set by converting image with cv.dnn.blobFromImage, but how to convert name: "seq_ind" for setInput ?

I know it has been really long....I hope you have got the answer, but if you haven't here it is:

Instance Methods

infer(inputs=None)
Description: Starts synchronous inference for the first infer request of the executable network and returns output data. Wraps infer() method of the InferRequest class
Parameters:
**inputs - `A dictionary` that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer**
Return value: A dictionary that maps output layer names to numpy.ndarray objects with output data of the layer

    Usage example:
    >>> net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
    >>> plugin = IEPlugin(device="CPU")
    >>> exec_net = plugin.load(network=net, num_requsts=2)
    >>> res = exec_net.infer({'data': img})
    >>> res
    {'prob': array([[[[2.83426580e-08]],
    [[2.40166020e-08]],
    [[1.29469613e-09]],
    [[2.95946148e-08]]
    ......
    ]])}

[n-mam]

Just for example,

import cv2 as cv

net = cv.dnn.readNet('/opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.bin',
                     '/opt/intel/computer_vision_sdk_2018.4.420/deployment_tools/intel_models/face-detection-adas-0001/FP32/face-detection-adas-0001.xml')

cap = cv.VideoCapture(0)

while cv.waitKey(1) < 0:
    hasFrame, frame = cap.read()
    if not hasFrame:
        break

    blob = cv.dnn.blobFromImage(frame, size=(672, 384))
    net.setInput(blob)
    out = net.forward()

    for detection in out.reshape(-1, 7):
        confidence = float(detection[2])
        xmin = int(detection[3] * frame.shape[1])
        ymin = int(detection[4] * frame.shape[0])
        xmax = int(detection[5] * frame.shape[1])
        ymax = int(detection[6] * frame.shape[0])

        if confidence > 0.5:
            cv.rectangle(frame, (xmin, ymin), (xmax, ymax), color=(0, 255, 0))

    cv.imshow('OpenVINO face detection', frame)

Do we have a c++ dnn sample which works with openvino model files ? I am facing similar issue as highlighted in ticket #2567