Inferencing - speed vs accuracy?

[lunaticds]

Hi All,
Short back story: Frigate running a Proliant DL380 Gen10 with an Intel Arc A310 GPU. Hardware doesn't support re-BAR so the GPU isn't running as well as it should. Using Yolov9s @ 320x320, my inferencing times are around 20ms with around 15 cameras at home.

I note that Blake mentioned that Yolov9 support for Coral TPUs is about to hit stable. I happen to have one floating around.

Is it worth going back to the Coral to get my inference speed down (I've had it down to around 10ms on that server) but take the slight hit on accuracy, or just deal with the slower 20ms inference times from the A310?
I haven't really gotten my head around whether the inferencing speed is going to matter that much.

[inkeep-ai-assistant[bot]]

Your 20ms inference time with the Intel Arc A310 using YOLOv9s at 320x320 is actually quite reasonable(1). The documentation shows that OpenVINO (which your A310 uses) with YOLOv9 at 320x320 typically achieves around 4.8ms on newer Intel Arc hardware, but your performance is affected by the lack of re-BAR support(2).

Understanding Detector Capacity:

Your detector capacity determines how many detections per second your system can handle(3). With 20ms inference speed, you can process approximately 50 detections per second (1000ms / 20ms = 50)(3). Whether this matters depends on how many motion regions your 15 cameras generate simultaneously(3).

Coral TPU vs Intel Arc A310:

The Coral EdgeTPU supports both the default Mobiledet model and YOLOv9 models(1). YOLOv9 models compiled for TensorFlow Lite are supported on Coral, with the edgetpu variant specifically designed for Google Coral compatibility(4)(5). The YOLOv9 edgetpu variant would give you better accuracy than the default Mobiledet model while maintaining faster inference speeds than your current A310 setup(4).

When Inference Speed Matters:

According to the documentation, you should consider adding detector capacity when "Skipped FPS is consistently > 0 even during normal activity"(3). If you're not seeing skipped frames, your current 20ms inference time is sufficient for your camera load(3).

Recommendation:

If you're not experiencing skipped detections or performance issues with your current setup, the 20ms inference time is acceptable(3). However, switching to the Coral with YOLOv9 edgetpu would give you both faster inference (~10ms as you mentioned) and improved accuracy over the default Mobiledet model(1)(4)(5). The YOLOv9 model is described as "more accurate but slower than default model" on Coral, but still faster than your current A310 performance(1).

📚 Sources:

1. Supported Hardware
2. GPU statistics for Intel xe kernel driver devices (Intel Arc B-series cards and optionally A-series)
3. High CPU Usage
4. Models
5. Requesting feedback on YOLOv9/EdgeTPU Google Coral Plugin

---

Have another question? Just tag @inkeep.