Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Expected inference time on the object detection model - >4s? #259
I did some tests and it take a little over 4 seconds to process a single frame with the built in object detection model. I realize it is not completely the same, but I was hoping to get performance somewhat like what Intel published for their Movidius Neural Compute stick using MobileNets here: https://software.intel.com/en-us/articles/mobilenets-on-intel-movidius-neural-compute-stick-and-raspberry-pi-3 .
Is this expected behavior? Is there a way to speed up inference with this build-in model?
Also, what does the
For reference, I used the
This was after I tried a similar procedure with 'CameraInference(object_detection.model())`, which is my real goal.
The implementation of the inference run itself for Movidius should be slightly faster than on the neural compute stick - we have written the inference engine from scratch, improving on runtime in certain places.
To understand what's happening here, you may need to know a bit more how the bonnet is wired. We have developed the board for the purpose of analyzing the images that are being captured by the camera. As such, we implemented a MIPI passthrough interface. It means that VisionBonnet is "listening in" on the image traffic that goes between the raspberry pi and PiCam2, and is capable of running inference on those pictures, without slowing down raspberry pi. In fact, raspberry pi does not even know that there's someone "listening in" on the pixel traffic between it and the camera.
However, for this to work we also need to have another channel of communication between the VisionBonnet and raspberry pi - to transmit the network file, and deliver the results of inference. Raspberry pi is very limited when it comes to fast buses, so we utilized SPI bus that takes a few of pins on the 40-pin header. This bus runs fairly slow in comparison to PCIe, USB3 or any other fast interface. But this is the only one available on the raspberry pi right now.
So in case of CameraInference() - the network is transmitted only once, after which we run inference on the pictures that are coming from the camera. However, in case of the example you have provided here, you actually transmit your image over the SPI bus to the Movidius chip on the Vision Bonnet each time you run the inference. And I am sure the majority of the time is taken by this transmission.
I am curious what is the resolution you use for your image? Have you tried timing the object_detection_camera.py example?
@PeterMalkin - thanks for the process flow explanation. Everything makes a lot more sense now.
In my static image tests, I was using an image from the
I did my tests again using
Here is my code:
Here is the output:
I am in the middle of an reinstall after downloading the repo updates from a few days ago - I'll give the new
The update just finished (cloned the repo, deleted
It looks like some improvements were made to the object detector model in the upgrade - now I am getting 2-3 FPS with the same code:
The python process is now running at 75-85% of the CPU, a little less than last time too.
pushed a commit
Feb 27, 2018
Chad, Thanks for bringing this to our attention. We improved our python side post processing to reduce latency to 155 ms per inference. This also reduces CPU usages. We are working on an improved protocol to reduce latency, it will be available at later release.
@weiranzhao - thanks for the update. I just loaded your recent commit. Using my code above, I can confirm I see a faster inference inline with your measurement:
@weiranzhao, @dmitriykovalev, @PeterMalkin - just for reference, here is a write-up on my project: https://webrtchacks.com/aiy-vision-kit-uv4l-web-server/
And a summary of the performance:
I still have some work to do to reduce CPU usage. When I run it for a while and the CPU heats up, the subsequent throttling causes a big delay but it works well before that point.
Thank you for your help here. I will close this issue unless you think there is value in keeping it open.