Object detection: bounding boxes drifted

[amararjun]

I am using an artpec-7 camera and trying object detection program. I made a custom MobilentSSDv2 model for detection.But, the output from the model running on the camera are different from the inference on an image taken with the camera.

I ran an inference using tensorflow tflite-interpreter on 1280X780pixel image resized to 320X320 pixels on an image taken from the camera. Meanwhile, the output log of the object detection app on the camera gives different results. The bounding boxes are drifted and sometimes, the detected classes are different. Few sample outputs are below: [ Left hand images are the result from tensorflow lite interpreter, right hand side images show the bounding boxes from object detection plotted on the image]

I am focusing on better localization of bounding boxes. Is this difference in the outputs normal ? Can I get better fitting bounding boxes on larod as that from the tflite-interpreter ?

[pataxis]

Hi @amararjun , thanks for your question, we'll get back to you when we had time to look in to this.

[Corallo]

Hello @amararjun

Which chip of the axis device are you using? Is the model running on CPU or on TPU?
Are you comparing the "pure" tflite lite model with a version converted for edgeTPU?

If the model is exactly the same, the output should be equal as well, but if you are using the TPU, going to the process of conversion to edgeTPU it is possible that the model looses some precision there.

An alternative, is that there is something wrong in the way the output of the model is interpreted to generate the boxes, you should try to isolate these scenarios and test them one by one.

Here I am going to assume that you are trying to run the model for edgeTPU.

• First take a camera frame, and run an inference with the model loaded on the tflite interpreted on your computer and on the CPU of the axis camera.
  You can extract the two output and compare them, the mean difference byte wise should be <= 2.

• If that works well, convert the model to edgeTPU, and run the same experiment, but running the model on the TPU of the camera, and compare the output bytes.

• If that also works well, the problem might be in the output interpretation.

If that help, to try your model on the camera you can SSH into it, and run:
larod-client -R 1 -p -c <CHIP> -g <model_path> -i <input_image_binary_format> -o /tmp/output.bin
This will produce a .bin file with the output of your model, you will have several files if the model has several output layers

Here's an example of how you could do it.
This python script assume a model that has only one output layer, you can use it as inspiration or modify it to make it work with your model output.
compare_larod_execution.txt

[amararjun]

Thank you for the comprehensive reply!
I run the model on the CPU of the camera and am using a pure .tflite model.I will go through the steps and analyse the outputs to check the differences.Hope to figure out, soon.

[Corallo]

If that is the case, you can skip the test on the EdgeTPU for now, check only if the output of the model is the same (or similar enough) in the camera and your tflite interpreter given a fixed input.
If that is not the case, we might have to investigate more.

[amararjun]

Hello @Corallo , Thank you for sharing the code and pointing out the possibilities.

1. I saved the input buffer to the larod model, and did inference on the tflite interpreter. The bounding box output was almost same.
2. The input video stream which is resized and fed to the larod model was chosen to be 480X360 ,automatically ,in the program.The aspect ratio for that stream in P1445E is different from that of 1920X1080 video stream which is used for the cropping the detections. Also, the field of view is slightly truncated for 480X360 . These might have resulted in the drifted bounding boxes.

I now have two options : to feed 1920 X 1080 stream to the larod model and get better fitted bounding boxes or use the 480 X 360 image and get smaller but better fitted bounding boxes

[Corallo]

Makes sense.
When you use the function imgprovider you get a provider that is going to provide an image with shape as close as possible to what you requested.
There are some fixed image shapes that you can get, you can easily find them in the camera GUI -> stream -> General. Some of them are 4:3, some 16:9 and some 16:10.
If you want the full field of view, I think you should go for a 16:9 frame.
You could pick the 640x360 frame rate for the model and the 1920x1080 for the frame to display.
Then, the best would be to letterbox the image, adding black bands on the top and the bottom, to get to the shape ratio expected from your model, and then rescale to the right model size.
Then, you are going to take this into account when you calculate box the position of the bounding box in the high resolution image.

[amararjun]

Thank you for the ideas! It did help!