loading data into InferenceModel

[liviolima80]

Dear all,
I tried to modify InferenceModel in order to run a CNN network on single image. The problem is that if I compare the output of the network with the same model using Caffe library I get different output. I guess that the problem could be related to how the image data are loaded into input vector for the net.

This is how i done, starting from OpenCV image:

cv::Mat image = cv::imread("img.png", CV_LOAD_IMAGE_COLOR);
cv::Mat image_float;
image.convertTo(image_float, CV_32FC3);
image_float = image_float / 255.0;
std::vector<float> input_array;
input_array.assign((float*)image_float.datastart, (float*)image_float.dataend);

    ...........

    InferenceModel<armnnCaffeParser::ICaffeParser, float> model(params);
    std::vector<float> output(model.GetOutputSize());
    model.Run(input_array, output);

Can anyone give me some feedback about this or explain me how the image data has to be organized into input_vector?

Regards

[MatthewARM]

I'm afraid I'm no expert on OpenCV but I think CV_32FC is NHWC, i.e. in memory the data for an RGB image is RGB,RGB,RGB,etc..,

Caffe convolution needs NCHW, in the Caffe documentation this is described as "n * c_i * h_i * w_i" here: http://caffe.berkeleyvision.org/tutorial/layers/convolution.html. I.e. it's a plane of all the R, followed by all the G, then all the B.

So before passing the image data to your model, as well as converting the pixels to 32-bit floating point, you also have to rearrange it into NCHW layout.

For ExecuteNetwork this is done in InferenceTestImage.cpp.

Hope that helps,

Matthew

[liviolima80]

Hi @MatthewARM ,
actually you are right. I changed the organization of input data, but the issue still remains. With further investigation I found that the problem is related to the caffe model. Let you consider the following caffe models, that represent a convolutional net with 3 output classes.

• model 1: caffe model generated by default Python training
• model 2: following the suggestion of @TelmoARM in my old thread (problem running ExecuteNetwork test #7) is generated in Python as
  import caffe
  net = caffe.Net('model1.prototxt', 'model1.caffemodel', caffe.TEST)
  net.save('model2.caffemodel')

The following are the results for using both the models with a) standard Caffe c++ library and b) armnnCaffeParser with the same input image

a) standard Caffe c++

• model 1: output probabilities : [1 4.49625e-12 2.43979e-18] -> correct
• model 2: Armnn Error: Unsupported layer type 'Data' (as reported in old issue 7)

b) armnnCaffeParser

• model 1: output probabilities : [0.471663 0.248880 0.279457] -> wrong
• model 2: Armnn Error: [0.471663 0.248880 0.279457] -> wrong

So actually it seems that caffe library and armnn work in the same way but there is still something related to the model. If needed I can provide both the models

[MatthewARM]

I'm pretty sure the problem must still be in the processing of the input data. Are you able to run your model through ExecuteNetwork?

[liviolima80]

Hi @MatthewARM ,
I don't think the problem is in the input data since I'm running in the same program the code for Caffe library C++ and for ArmnnCaffeParser. This is what I do:

• First I build the model for Caffe c++, load the data and run it. I definetly sure that in the Caffe c++ model input data are correctly loaded since I'm succesfully working with it from a while
• Than I take the vector of input data from Caffe model and I put it as input to Armnn model, I just need to convert from float* to vector

Sorry I've just see that my last msg was wrong. This is the right situation for the results:

a) standard Caffe c++

• model 1: output probabilities : [1 4.49625e-12 2.43979e-18] -> correct
• model 2: output probabilities : [0.471663 0.248880 0.279457] -> wrong

b) armnnCaffeParser

• model 1: Armnn Error: Unsupported layer type 'Data' (as reported in old issue 7)
• model 2: output probabilities: [0.471663 0.248880 0.279457] -> wrong

As you can see for model2 Caffe c++ and Armnn give the same wrong results

[MatthewARM]

If you are getting the same results from Caffe and Armnn, then the problem must be in the input data. Presumably the old 'data' layer was doing some stuff automatically that is not done by the new 'input' layer.

[liviolima80]

Ok @MatthewARM ,
I tried to open an issue on Caffe official repository since I don't have an idea on what has to be modified.

Nevertheless I think that more documentation and example data have to be provided on Armnn repository, since at the moment it is not easy to understand how to correctly use the library

[MatthewARM]

Hi @liviolima80,

I agree and we're working on new documentation just as much as we're working on new functionality and performance.

On the subject of examples, in this case the ArmNN runtime is performing as intended as a drop-in replacement for the Caffe runtime, so you should be able to follow a Caffe example. There is also code at https://github.com/ARM-software/ML-examples which has an example use of ArmNN in an MNIST application.

[MatthewARM]

HI @liviolima80 I'm closing this issue as it seems the original problem (data layout) has been resolved. if you figure out what the remaining problem is then please do let us know.