Frigate 0.17 custom model for rk3588 (rknn)

[sstepane]

Hi,

After being hitting a wall for couple days (I'm not very proficient with models) still struggling to provide custom rknn model for frigate on rk3588 (orange pi 5).
I've tried various tips and tricks and best I could get is converted yolov8s (ultralytics) to onnx to rknn which kinda works, but not sure if it works properly, since there are weird detections, etc. On onnx-rknn conversions I get unknown ops. I suppose model is not functioning properly due to those warnings/errors during onnx-rknn conv.
So, I'm trying to figure out a way for a clean yolo (8, 9, may be x) of small model to properly export to onnx and then to rknn with no warning on unknown ops but has found no way yet. There're so many quirks like including recommended rknn zoo lab for fixing yolo models for seamless rknn converstion... But their models do not fit frigate either (yolov8s for sure).
Is there a proper pathway for this chain? I see yolov8s even broken gives quite good performance, so, why stick with nano/tiny.
Why am I not satisfied with ready models? I need particulars classes (and, in fact, only 4 of them - so I had to add 75 dummy ones, since frigate parser need 80...).

[inkeep-ai-assistant[bot]]

For Rockchip RKNN models on Frigate 0.17, there's a documented pathway for converting your own ONNX models to RKNN format. (1)

Conversion Process

Place your ONNX model(s) in config/model_cache/rknn_cache/onnx on your Docker host, then create a configuration file at config/conv2rknn.yaml: (1)

soc: ["rk3588"]
quantization: false

output_name: "{input_basename}"

config:
  mean_values: [[0, 0, 0]]
  std_values: [[255, 255, 255]]
  quant_img_RGB2BGR: true

(1)

Run the conversion with: (1)

docker exec <frigate_container_id> python3 /opt/conv2rknn.py

(1)

The converted models will be placed in config/model_cache/rknn_cache. (1)

Supported Model Types for RKNN

The documentation shows these model types work with RKNN: (1)

• YOLOv8 with model_type: yolov8
• YOLOv9 with model_type: yolo-generic
• YOLOx with model_type: yolox

Custom Labels Issue

Regarding your 4-class requirement with 75 dummy classes - this is a constraint you're hitting. The documentation references labelmap files like /labelmap/coco-80.txt for 80-class models. (1)

Frigate+ Alternative

If you have a Frigate+ subscription, YOLOv9 models are automatically converted for Rockchip as of 0.17. (2) You can also request custom-trained models, though you'd need to verify if custom class subsets are supported through that workflow.

For the "unknown ops" warnings during ONNX-to-RKNN conversion, this typically indicates operations in your ONNX export that aren't supported by the RKNN toolkit. The documented conversion process using Frigate's built-in script should handle compatible models properly. (1)

📚 Sources:

1. Supported Hardware
2. Models

---

Have another question? Just tag @inkeep.

[NickM-27]

what is your config?

[sstepane]

Model yolov9s (original, not ultralytics)
simple onnx export

conv2rknn
soc: ["rk3588"]
quantization: false

output_name: "best"

config:
mean_values: [[0, 0, 0]]
std_values: [[255, 255, 255]]
quant_img_RGB2BGR: true

frigate config of model

model:
  path: /config/model_cache/rknn_cache/onnx/best.onnx
  #model_type: yolox
  #path: /config/model_cache/rknn_cache/best_320_4.rknn
  model_type: yolo-generic
  width: 320
  height: 320
  input_tensor: nhwc
  input_pixel_format: rgb
  labelmap_path: /config/best_320_labels.txt
  #labelmap_path: /labelmap/coco-80.txt
  labelmap:
    0: drone
    1: person
    2: vehicle
    3: weapon

[NickM-27]

it seems most likely this was just not exported in the way that it is expected https://docs.frigate.video/configuration/object_detectors#yolov9-for-other-detectors

[sstepane]

Tried this export. Got same ValueError: not enough values to unpack (expected 4, got 3)... I've trained yolov9 exactly on this version (https://github.com/wongkinyiu/yolov9). Only difference was with "--simplify" option, but still no go...

[NickM-27]

I don't know why this would happen, it is falling into the multipart yolo branch meaning it has more outputs than it should.

[sstepane]

there might be difference... it seems I've trained original model, but snippet is about "converted" version (supposedly newer ultralytics variant). have to try that. (train converted and then export-export).

[sstepane]

Still can't get it to work.
Is converting scripts in frigate modify head of onnx model during conversion to rknn or it has to be already fully compatible with rknn (possibly as described in rknn model zoo?)?
As far as I understand - it has to be not only compatible with rknn (no unknown ops during conversion) but also meet frigate expected head, which is, supposedly exactly 3 tensors structured in a specific shape format: (1, 3, 85, 80, 80), (1, 3, 85, 40, 40), and (1, 3, 85, 20, 20)...

[NickM-27]

As far as I understand - it has to be not only compatible with rknn (no unknown ops during conversion) but also meet frigate expected head, which is, supposedly exactly 3 tensors structured in a specific shape format: (1, 3, 85, 80, 80), (1, 3, 85, 40, 40), and (1, 3, 85, 20, 20)...

right, that's the expected shape for the single part YOLO handler.