Getting slow inference times on N100 with YOLO-NAS and YoloV11

[devedse]

Hey all,

I'm currently trying to setup Frigate with 3 camera's. I have an N100 Intel CPU with iGPU which I want to use. Everything is running inside an LXC container with the iGPU mapped to the container. Within this LXC container I run a docker container which again maps the iGPU.
The problem is that I'm only getting around 150ms inference times though. I've seen other users post results around 25ms.
Some references:

• [Detector Support]: Just got OpenVino capable N100. Any reason to keep using Coral? OpenVino vs Coral? #11717 (reply in thread)
• [Detector Support]: Yolov8 use in 0.14 Beta 3 #12011 (reply in thread)

Here's the output of intel_gpu_top on the host:

It does seem to use the iGPU (at least for FFMPEG)

Here's a screenshot of the System Metrics page:

My docker-compose.yml looks as follows:

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.16.0-beta1
    shm_size: "640mb" # update for your cameras based on calculation above
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /root/dockercomposers/frigate/config:/config
      - /root/dockercomposers/frigate/storage:/media/frigate
      - /root/dockercomposers/frigate/models:/models
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "***********"
      #LIBVA_DRIVER_NAME: "i965"
    cap_add:
      - CAP_PERFMON

And my frigate config:

ffmpeg:
  #hwaccel_args: preset-vaapi
  hwaccel_args: preset-intel-qsv-h264

detectors:
  ov1:
    type: openvino
    device: GPU

#YOLONAS
model:
  model_type: yolonas
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  path: /models/yolo_nas_s.onnx
  labelmap_path: /models/coco_80cl.txt


# model:
#   model_type: yolo-generic
#   width: 640
#   height: 640
#   input_tensor: nchw
#   input_dtype: float
#   input_pixel_format: bgr
#   path: /models/yolov11/yolo11n.onnx
#   labelmap_path: /models/coco_80cl.txt

objects:
  track:
    - person
    - dog
    - cat
    - car
    - motorcycle
    - bird
    - bicycle
    - vehicle registration plate
    - woman
    - man
    - mouse
    - glasses

cameras:

  G4Bullet:
    ffmpeg:
      inputs:
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - detect
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: true
    record:
      enabled: true
      alerts:
        retain:
          days: 5
      detections:
        retain:
          days: 5

  G4Doorbell:
    ffmpeg:
      inputs:
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - detect
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - record
    detect:
      width: 960
      height: 720
      fps: 5
    snapshots:
      enabled: true
    record:
      enabled: true
      alerts:
        retain:
          days: 5
      detections:
        retain:
          days: 5

  G5TurretUltra:
    ffmpeg:
      inputs:
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - detect
        - path: rtsps://10.88.10.253:7441/***************?enableSrtp
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: true
    record:
      enabled: true
      alerts:
        retain:
          days: 5
      detections:
        retain:
          days: 5

version: 0.16-0
detect:
  enabled: true

Does someone know what could be impacting the performance?

[devedse]

I did a reboot and now the INTEL_GPU_TOP stats look a bit more detailed:

In addition the GPU stats work again on the Frigate page:

[xavan83]

You can export yolo11 with openvino format using int8 precision (and disable nms as Frigate uses its own)

[devedse]

But that doesn't really explain why other people are getting 25ms inference times with the YOLO-NAS model. I assume they've just downloaded that from the notebook linked on the Frigate docs.

[NickM-27]

yeah something seems odd here