Discussion comparing coral and iGPU (OpenVino) as detector, pros & cons

[quenthal]

I've frigate running on minipc with Iris XE iGPU. I 've also Coral USB and Coral mPCI available.

I've been comparing these within Proxmox LXC, where Frigate container is running. 7 cameras.

I get interference slight below 8ms with mPCI, USB version nearly the same. With OpenVino with iGPU I get interference around 10-12ms.

With OpenVino CPU on LXC with 4 virtual cores CPU usage (looking from Proxmox) is generally around 12-13% with recurring spikes up to 40%.

With Coral the cpu is generally also around 12-13%, but the spikes are slightly less recurring, up to 32%, sometimes up to 40%.

Other points to compare are of course the quality of the detecting between OpenVino ja Coral. I haven't yet got any definitive data on this.

One big, major I would say thing to compare is also power consumption. I haven't yet unplugged for these comparisons mPCI Coral physically, but it would be also intresting to see if Coral consumes more power than what the increased usage of iGPU and maybe CPU causes.

What are your findings or thoughts on these matters?

[NickM-27]

Coral is more power efficient. Their accuracy also has nothing to do with the device itself as that is only dependent on the model

[quenthal]

Thanks!

[bazylhorsey]

I think that is a somewhat technically correct assessment @quenthal. However, there are a lot of limitations on models that can be ran on coral due to hardware limitations. In computer vision specifically, it's really hard to get results you see that are crazy accurate like yolo v8.
I'd be curious to see some kind of accuracy/latency graph for edge Intel/coral accessible models. I'm trying to create custom models that are both very accurate, and very fast. This proves difficult in an edge ecosystem, at the same time, these models are impossible to run anywhere else in the same cost point.

[gszigethy]

I have tested a few setups and the difference is actually not that big between Coral and OpenVino which is a big surprise for me too.

I have 9 cameras monitored, the detect streams are 1280x720, the record streams are various, but between FHD and 4k, all of them using h.264. Storage backend in each case is a CIFS mounted share on the same switch with an SSD backend, all ports are 1Gb. Cpufreq governor set to ondemand, powertop auto-tune.

• Raspberry Pi 4, USB Coral as detector: 9W (including ~1w of a POE splitter and a tiny fan in the case): CPU 40-50%, inference speed 20-30 ms
• NUC8i5BEK, Proxmox 7, Frigate in a docker container on a Debian LXC, GPU passthrough, OpenVino: 25-35W, CPU 20-30%, GPU 10%, inference 7-9 ms
• NUC11TNHv50L, Proxmox 7, Frigate in a docker container on a Debian LXC, GPU passthrough, OpenVino: 16-26W, CPU 20-30%, GPU 10%, inference 7 ms
• NUC11TNHv50L, Proxmox 7, Frigate in a docker container on a Debian LXC, GPU passthrough, USB Coral: 20-32W, CPU 7-15%, GPU 2%, inference 7-8 ms

Some observations:

• Pi is the lowest power by far and can handle the above number of cameras, but tested only with Frigate 11 (fr 12 seem to had various issues on pi, didn't have the time to debug yet).
• Pi, while low power and can handle the NVR capabilities, watching events and using the interface in general is not as snappy as a NUC, obviously.
• Accuracy of OV and Coral detection are very similar. I've compared a day of events on the most busy camera (facing a street) and it was almost identical. They mostly miss events nighttime when the camera picture is worse, lower fps, the screen is blurry, etc. But the records are still there as the movement is detected, just not detected as an event (so you can still capture it, but less convenient to check what happened).
• Confidence of OV is much higher, ie. OV events are in the high 90%, Coral is more distributed in the range of 60-100%.
• The power consumption is stable on the pi, 9w, probably because it goes on max fequency already at 50% cpu usage. The power draw of the NUCs can double based on amount of movement and event detection. For instance, on a windy day can go significantly higher.
• Using poor quality cable for the Coral and connecting it on USB2 (480M in lsusb) makes the inference speed and the CPU usage go kaboom.

In general I'm very much pleased with the OpenVino on a NUC, especially if there are Coral TPU shortages, it's a perfectly viable route, also considering that ffmpeg on amd64 is much less picky then on a pi. It eats more power, but you get a lot more too.

[NickM-27]

Just wanted to note

• Accuracy of OV and Coral detection are very similar. I've compared a day of events on the most busy camera (facing a street) and it was almost identical.
• Confidence of OV is much higher, ie. OV events are in the high 90%, Coral is more distributed in the range of 60-100%.

These are factors of the model being run and have nothing to do with the hardware that is used for detection (coral, OV, etc.)

[mac587]

Am I reading this correctly? That coral is basically redundant on anything with intel iGPU from 6th gen onwards?

[cdennis12280]

Think so I'm running on old dell laptop i5 6tg gen. 4 cams, inf speed around 12-15ms.

Now need to source a cheap tiny pc with decent cpu and proxmox my setup.