Opencv DNN Inference Engine FPGA Target

[balachandarsv]

Tried using the FPGA Target in Inference Engine using OPENCV Dnn. Getting the following error.
I tried setting the library path as LD_LIBRARY_PATH still the same error.

opencv-4.1.2/modules/dnn/src/op_inf_engine.cpp:704: error: (-215:Assertion failed) in function 'initPlugin'

Failed to initialize Inference Engine backend: Failed to create plugin /opt/intel/openvino_2019.3.376/deployment_tools/inference_engine/lib/intel64/libdliaPlugin.so for device FPGA
Please, check your environment
Cannot load library '/opt/intel/openvino_2019.3.376/deployment_tools/inference_engine/lib/
]

The same model works on CPU Target.

How to solve this issue?

[dkurt]

@balachandarsv, Hi!

Do you use FPGA distribution of OpenVINO?

[scianand]

Hi

[balachandarsv]

@dkurt Yes. I am using FPGA Distribution only. The required .so file libdliaPlugin.so is available at that location. But still I am getting the same error while calling net.forward()

[alalek]

Cannot load library

Running LD_DEBUG=libs <your_app> should help with investigation.

[balachandarsv]

I was able to solve this by keeping the libs in intel fpga sdk folder.

But while running the googlenet model (bvlc_googlenet.caffemodel) I am getting the following error.

How to activate heterogeneous plugin of InferenceEngine from opencv? @dkurt @alalek

OpenCV(4.1.2) /opt/opencv-4.1.2/modules/dnn/src/op_inf_engine.cpp:704: error: (-215:Assertion failed) in function 'initPlugin'

Failed to initialize Inference Engine backend: Graph is not supported on FPGA plugin due to existance of layer (Name: pool1/norm1, Type: Norm)
in topology. Most likely you need to use heterogeneous plugin instead of FPGA plugin directly.
]

[dkurt]

@balachandarsv, Please check the following code execution:

opencv/modules/dnn/src/op_inf_engine.cpp

Lines 742 to 765 in 8108fb0

	#if INF_ENGINE_VER_MAJOR_LE(INF_ENGINE_RELEASE_2019R1)
	plugin = InferenceEngine::InferencePlugin(enginePtr);
	netExec = plugin.LoadNetwork(net, {});
	#else
	bool isHetero = false;
	if (device_name != "CPU")
	{
	isHetero = device_name == "FPGA";
	for (auto& layer : net)
	{
	if (layer->type == kOpenCVLayersType)
	{
	layer->affinity = "CPU";
	isHetero = true;
	}
	else
	layer->affinity = device_name;
	}
	}
	if (isHetero)
	netExec = ie.LoadNetwork(net, "HETERO:" + device_name + ",CPU");
	else
	netExec = ie.LoadNetwork(net, device_name);
	#endif

[balachandarsv]

@dkurt I tried the same with the above commit, still the same response.

Most likely you need to use heterogeneous plugin instead of FPGA plugin direct

Is there any variable that i should set in addition

[dkurt]

@balachandarsv, I just wanted to ask, which code part is used (2019R1 or another one) and which device name is used there ("CPU" or "HETERO:FPGA,CPU"). Please check it by print.

[balachandarsv]

@dkurt I am using opencv JAVA bindings. I will try to use the c++ and update here.

I am having the device as FPGA

Failed to initialize Inference Engine backend (device = FPGA): Graph is not supported on FPGA plugin due to existance of layer (Name: MobilenetV2/Logits/Conv2d_1c_1x1/act_quant/FakeQuantWithMinMaxVars/Transpose, Type: Permute)

in topology. Most likely you need to use heterogeneous plugin instead of FPGA plugin direct
]

I even tried for mobilenet model too.

[balachandarsv]

@dkurt Device name : FPGA , ishetero : true . Its branching to the else part, not to 2019R1.

Device name is not changing to HETERO:FPGA,CPU.

It's still FPGA.

I printed these in the catch block in that function.

I am facing error in this line

netExec = ie.LoadNetwork(net, "HETERO:" + device_name + ",CPU");

[dkurt]

@balachandarsv, thanks!

Can you please provide a model to reproduce it?

[balachandarsv]

Mobilenet Model :

https://download.01.org/opencv/2019/open_model_zoo/R2/20190628_180000_models_bin/mobilenetv2-int8-tf-0001/FP16/

Googlenet Model : https://github.com/intel/caffe/tree/master/models/bvlc_googlenet

You can use either of these models.

I just used model optimizer to convert the googlenet model to FP16 for bin and xml files so that i can run using IE.

@dkurt let me know if you need the googlenet bin and xml. I will attach them too.

[ilya-lavrenov]

Could you please run LoadNetwork for HETERO:FPGA,CPU device with additional flags HETERO_CONFIG_KEY(DUMP_GRAPH_DOT) -> YES?

It forces HETERO plugin to dump .dot files which shows how HETERO plugin splitted layers between CPU and FPGA devices.