Using a better model for detection

[Honusnap]

Hi all,
Yesterday i've installed Frigate as LXC on a proxmox server with an already installed Home assistant.
All seems fine but i would like a better detection performance then the stock /openvino-model/FP16/ssdlite_mobilenet_v2.xml.

Here is the interesting part of my Frigate config file : i was using this config at start with good performance on my i7100U NUC :
detectors:
ov:
type: openvino
device: GPU.0
model:
width: 300
height: 300
input_tensor: nhwc
input_pixel_format: bgr
path: /openvino-model/FP16/ssdlite_mobilenet_v2.xml
labelmap_path: /openvino-model/coco_91cl_bkgr.txt

Then i downloaded this model : https://docs.openvino.ai/2024/omz_models_model_mobilenet_v2_1_0_224.html and converted it with the command : omz_converter -o . --name mobilenet-v2-1.0-224
Which create the /openvino-model/FP16/ssdlite_mobilenet_v2.xml file

I then modified my frigate config to :

detectors:
ov:
type: openvino
device: GPU.0
model:
width: 224
height: 224
input_tensor: nhwc
input_pixel_format: bgr
path: /openvino-model/mobilenet-v2-1.0-224.xml
labelmap_path: /openvino-model/labelmap.txt

And i got this error :
2024-10-09 17:38:55.386428 [2024-10-09 17:38:55] frigate.detectors.plugins.openvino ERROR : SSD model output doesn't match. Found [1,1001]
What is it supposed to mean ? I checked the piwxel size that seems to be 224 ... I'm a noob regarding frigate and all those models...
Maybe there is a better way, somewhere to find a pre learned model that would be better then the stock frigate one, can't find a proper yolonas xml file...
I'm living in the country side, i do not need to detect cars or things like that but animals like fox, cat, dogs... etc (goal is to protect my chickens) and still be able to detect humans, and even better faces to not raise alarm if the detected people is a family member.

Here is the frigate error log :

2024-10-09 17:38:46.471562 [INFO] Preparing Frigate...
2024-10-09 17:38:46.471660 [INFO] Starting Frigate...
2024-10-09 17:38:48.458026 [2024-10-09 17:38:48] frigate.app INFO : Starting Frigate (0.14.1-)
2024-10-09 17:38:48.458146 [2024-10-09 17:38:48] frigate.util.config INFO : Checking if frigate config needs migration...
2024-10-09 17:38:48.532431 [2024-10-09 17:38:48] frigate.util.config INFO : frigate config does not need migration...
2024-10-09 17:38:48.799803 [2024-10-09 17:38:48] frigate.util.services INFO : Automatically detected vaapi hwaccel for video decoding
2024-10-09 17:38:48.830890 [2024-10-09 17:38:48] peewee_migrate.logs INFO : Starting migrations
2024-10-09 17:38:48.831308 [2024-10-09 17:38:48] peewee_migrate.logs INFO : There is nothing to migrate
2024-10-09 17:38:48.837968 [2024-10-09 17:38:48] frigate.app INFO : Recording process started: 39425
2024-10-09 17:38:48.845427 [2024-10-09 17:38:48] frigate.app INFO : Recording process started: 39428
2024-10-09 17:38:48.850656 [2024-10-09 17:38:48] frigate.app INFO : go2rtc process pid: 126
2024-10-09 17:38:48.908378 [2024-10-09 17:38:48] detector.ov INFO : Starting detection process: 39482
2024-10-09 17:38:48.909504 [2024-10-09 17:38:48] frigate.app INFO : Output process started: 39492
2024-10-09 17:38:49.012208 [2024-10-09 17:38:48] frigate.app INFO : Camera processor started for AtelierFacade: 39536
2024-10-09 17:38:49.013947 [2024-10-09 17:38:49] frigate.app INFO : Camera processor started for AtelierHaut: 39546
2024-10-09 17:38:49.062133 [2024-10-09 17:38:49] frigate.app INFO : Camera processor started for AtelierBas: 39550
2024-10-09 17:38:49.062211 [2024-10-09 17:38:49] frigate.app INFO : Capture process started for AtelierFacade: 39558
2024-10-09 17:38:49.096010 [2024-10-09 17:38:49] frigate.app INFO : Capture process started for AtelierHaut: 39571
2024-10-09 17:38:49.130521 [2024-10-09 17:38:49] frigate.app INFO : Capture process started for AtelierBas: 39576
2024-10-09 17:38:55.386428 [2024-10-09 17:38:55] frigate.detectors.plugins.openvino ERROR : SSD model output doesn't match. Found [1,1001].
2024-10-09 17:38:55.388664 Process detector:ov:
2024-10-09 17:38:55.389537 Traceback (most recent call last):
2024-10-09 17:38:55.389610 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-10-09 17:38:55.389652 self.run()
2024-10-09 17:38:55.389689 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-10-09 17:38:55.389724 self._target(*self._args, **self._kwargs)
2024-10-09 17:38:55.389755 File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2024-10-09 17:38:55.389790 detections = object_detector.detect_raw(input_frame)
2024-10-09 17:38:55.389825 File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2024-10-09 17:38:55.389859 return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-10-09 17:38:55.389891 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 160, in detect_raw
2024-10-09 17:38:55.390006 infer_request.infer(input_tensor)
2024-10-09 17:38:55.390054 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 132, in infer
2024-10-09 17:38:55.390085 return OVDict(super().infer(_data_dispatch(
2024-10-09 17:38:55.390111 RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:116:
2024-10-09 17:38:55.390136 Exception from src/inference/src/cpp/infer_request.cpp:66:
2024-10-09 17:38:55.390161 Check 'port.get_element_type() == tensor->get_element_type()' failed at src/plugins/intel_gpu/src/plugin/sync_infer_request.cpp:142:
2024-10-09 17:38:55.390186 [GPU] Mismatch tensor and port type: f32 vs u8
2024-10-09 17:39:00.576854 2024-10-09 17:38:55.390211
2024-10-09 17:39:00.576881 2024-10-09 17:38:55.390247
2024-10-09 17:39:00.576885 2024-10-09 17:38:55.390288
2024-10-09 17:39:03.869320 [2024-10-09 17:39:03] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for AtelierFacade. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-10-09 17:39:03.871571 [2024-10-09 17:39:03] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for AtelierHaut. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-10-09 17:39:03.873862 [2024-10-09 17:39:03] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for AtelierBas. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-10-09 17:39:09.478073 [2024-10-09 17:39:09] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
2024-10-09 17:39:09.494173 [2024-10-09 17:39:09] detector.ov INFO : Starting detection process: 39726

Thanks to anybody that would help,

[NickM-27]

Have you tried the yolonas model in the docs? https://docs.frigate.video/configuration/object_detectors#yolo-nas

[Honusnap]

Thanks for helping Nick, something is not that clear for me, i followed "this notebook" referenced in your link

https://github.com/blakeblackshear/frigate/blob/dev/notebooks/YOLO_NAS_Pretrained_Export.ipynb
and ... i completed the first command in my Frigate LXC :
! pip install -q super_gradients==3.7.1
That downloaded like 1Gb of data ....

Then i put those script lines :

from super_gradients.common.object_names import Models
from super_gradients.conversion import DetectionOutputFormatMode
from super_gradients.training import models

model = models.get(Models.YOLO_NAS_S, pretrained_weights="coco")

in a Go.A.py file and run it with python3 Go.A.py ... and got a bunch of errors.

Obviously i missed/do not understand something .. :D

[NickM-27]

all you do is click the green button next to each step and then the last step will download the .onnx file to your computer that you are running it on

[Honusnap]

all you do is click the green button next to each step and then the last step will download the .onnx file to your computer that you are running it on

Lol .. i stupidly did not clicked on "open in colab" and was let in the text version of the notebook in github .. :D

Allright, got the onnx file, change configuration in Frigate...

model:
  # width: 300
  # height: 300
  # input_tensor: nhwc
  # input_pixel_format: bgr
  # path: /openvino-model/FP16/ssdlite_mobilenet_v2.xml
  # labelmap_path: /openvino-model/coco_91cl_bkgr.txt
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  path: /openvino-model/yolo_nas_s.onnx
  labelmap_path: /openvino-model/coco-80.txt

and ..

2024-10-09 21:19:01.895062  [2024-10-09 21:19:01] frigate.detectors.plugins.openvino ERROR   : SSD model input doesn't match. Found [1,3,320,320].
2024-10-09 21:19:01.895166  [2024-10-09 21:19:01] frigate.detectors.plugins.openvino ERROR   : SSD model output doesn't match. Found [20,7].

MMmmmm ... i got 320 in my config, do not know what the 1,3 before those 320 mean....

[NickM-27]

you have to set the model_type as yolonas

[Honusnap]

yolonas

you have to set the model_type as yolonas

I'm so sorry to waste your time, i can't even replicate informations from a webpage .. forgot to copy lines....
Was doing something else in parallel and should not have.

Allright, it seems to work, i'm on an old i7100U NUC, was having 13ms inference speed (using GPU and FP16/ssdlite_mobilenet_v2.xml) , now having 34ms with yolo_nas_s.onnx and it also seems to have impact on CPU (something like 20% vs 15%).

Thanks again for your time; i now have to optimise some things in the config