[Detector Support]: OpenVino crash when using GPU

[jjak0b]

Describe the problem you are having

I have a Proxmox 8.2.4 installation running on an Intel N3350 CPUì and a LXC unprivileged Debian 12 container running Dcoker which runs a Frigate Container.

The issue is that OpenVino with GPU detection crash the Frigate container but if i set CPU in detector type won't crash.
Hardware acceleration works and is detected using intel_gpu_top in proxmox host. (drivers are already installed)

The Frigate configuration i provide in this issue is about a UVC camera passed to Frigate with USB passthrough, but the same issue happens also using an IP Camera.

--
Proxmox 8.2.4 with kernel 6.8.12-1-pve
Docker installed in LXC: 27.1.1

Version

0.14.0

Frigate config file

database:
  path: /db/frigate.db

mqtt:
  enabled: true
  host: 192.168.1.200
  port: 1883
  user: '{FRIGATE_MQTT_USER}'
  password: '{FRIGATE_MQTT_PASSWORD}'
  topic_prefix: frigate
  client_id: frigate

telemetry:
  # Optional: Configure system stats
  stats:
    amd_gpu_stats: false
    intel_gpu_stats: true
    network_bandwidth: true

go2rtc:
  rtsp:
    username: '{FRIGATE_MQTT_USER}'
    password: '{FRIGATE_MQTT_PASSWORD}'
  streams:
    uvc_camera: ffmpeg:device?video=0&video_size=640x480#video=h264#hardware=vaapi#raw=-pix_fmt yuv420p
    leovin_cam:
      - rtsp://admin:password@camera-leovin.lan/stream0
      - ffmpeg:leovin_cam#video=copy#audio=aac#hardware
  webrtc:
    candidates:
      - 192.168.1.202:8555
      - stun:8555
birdseye:
  enabled: true
  mode: motion

# works
detectors:
  ov:
    type: openvino
    device: GPU

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

record:
  enabled: true
  events:
    retain:
      default: 5
      mode: active_objects
    pre_capture: 5
    post_capture: 20

objects:
  track:
    - person
    - bicycle
    - motorcycle
  filters:
    person:
      threshold: 0.65
    bicycle:
      threshold: 0.65
    motorcycle:
      threshold: 0.65
    airplane:
      threshold: 0.70

snapshots:
  enabled: true
  timestamp: true
  bounding_box: true
  retain:
    default: 5

ffmpeg:
  hwaccel_args: preset-vaapi

cameras:

  Retro_UVC_Computer:
    enabled: true
    live:
      stream_name: uvc_camera
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/uvc_camera
          roles:
            - detect
            - record
      input_args: preset-rtsp-restream

    zones:
      external_door:
        coordinates: 640,480,105,480,113,0,640,0
        objects:
          - person
    detect:
      enabled: true
    record:
      enabled: true
    snapshots:
      enabled: true

version: 0.14

docker-compose file or Docker CLI command

docker run -d \
  --name frigateT \
  --restart=unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
  --device /dev/video0:/dev/video0 \
  --device /dev/dri:/dev/dri \
  --network=bridge \
  --privileged \
  --cap-add=CAP_PERFMON \
  --group-add video \
  --group-add "106" \
  --group-add "44" \
  --shm-size=512m \
  --device-cgroup-rule="c 189:* rmw" \
  --device-cgroup-rule="c 226:* rmw" \
  -v /mnt/NVR:/media:rw \
  -v /portainer/Files/AppData/Config/FrigateT:/config:rw \
  -v /etc/localtime:/etc/localtime:ro \
  -v /portainer/Files/AppData/Data/FrigateT/db:/db:rw \
  -e LIBVA_DRIVER_NAME=i965 \
  -e FRIGATE_MQTT_USER=user\
  -e FRIGATE_MQTT_PASSWORD=mqttpass\
  -e FRIGATE_RTSP_PASSWORD='rtsppass' \
  -e TZ=Europe/Rome \
  -p 5000:5000 \
  -p 8971:8971 \
  -p 8554:8554 \
  -p 8555:8555/tcp \
  -p 8555:8555/udp \
  ghcr.io/blakeblackshear/frigate:0.14.0

Relevant Frigate log output

Process detector:ov:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
    object_detector = LocalObjectDetector(detector_config=detector_config)
  File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
    self.detect_api = create_detector(detector_config)
  File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
    return api(detector_config)
  File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
Exception from src/inference/src/dev/plugin.cpp:59:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:178:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:433:
[GPU] clWaitForEvents, error code: -14

Relevant go2rtc log output

...

[2024-08-27 23:16:03] peewee_migrate.logs            INFO    : Starting migrations
[2024-08-27 23:16:03] peewee_migrate.logs            INFO    : There is nothing to migrate
[2024-08-27 23:16:03] frigate.app                    INFO    : Recording process started: 286
[2024-08-27 23:16:03] frigate.app                    INFO    : Recording process started: 288
[2024-08-27 23:16:03] frigate.app                    INFO    : go2rtc process pid: 99
[2024-08-27 23:16:03] detector.ov                    INFO    : Starting detection process: 315
[2024-08-27 23:16:03] frigate.app                    INFO    : Output process started: 317
[2024-08-27 23:16:03] frigate.app                    INFO    : Camera processor started for Retro_UVC_Computer: 332
[2024-08-27 23:16:03] frigate.app                    INFO    : Capture process started for Retro_UVC_Computer: 335
[INFO] Starting go2rtc healthcheck service...
[2024-08-27 23:16:07] frigate.util.services          ERROR   : Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied)
Process detector:ov:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
    object_detector = LocalObjectDetector(detector_config=detector_config)
  File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
    self.detect_api = create_detector(detector_config)
  File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
    return api(detector_config)
  File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
Exception from src/inference/src/dev/plugin.cpp:59:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:178:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:433:
[GPU] clWaitForEvents, error code: -14

[2024-08-27 23:16:24] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...

...

Operating system

Proxmox

Install method

Docker CLI

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

N/A

Any other information that may be helpful

Proxmox unprivileged container lxc.conf

arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: dockero
memory: 5120
mp0: filedata:100/vm-100-disk-1.raw,mp=/portainer,size=9G
mp1: /mnt/net_shared/Videos,mp=/mnt/Videos
mp2: /mnt/net_shared/NVR,mp=/mnt/NVR
mp3: /mnt/net_shared/Music,mp=/mnt/Music
mp5: /mnt/net_games,mp=/mnt/Games
nameserver: 192.168.1.254
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.254,hwaddr=E2:2C:70:45:F5:A2,ip=192.168.1.202/24,type=veth
onboot: 1
ostype: debian
rootfs: filedata:100/vm-100-disk-0.raw,size=21G
startup: up=1
swap: 1024
unprivileged: 1
lxc.cgroup.memory.swappiness: 1
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.cgroup2.devices.allow: c 81:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/bus/usb/001/002 dev/bus/usb/001/002 none bind,optional,create=file
lxc.mount.entry: /dev/video0 dev/video0 none bind,optional,create=file
lxc.mount.entry: /dev/video1 dev/video1 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 104 1
lxc.idmap: g 107 100107 65429

[hawkeye217]

See the FAQ, Upgrade issues question 6.

[jjak0b]

Sorry i missed it. But i specify also that Frigate didn't work with GPU Detection also In 0.13 probably also caused by #12266

[NickM-27]

have you tried the fixes listed there?

[Jonoox]

So what can we do about this? I'm on Pentium J5005 and having same problem with GPU on OpenVino and frigate 14.1

[Jonoox]

I have weaker CPU and I'm getting 80-160ms of inference time.

[Jonoox]

With ssdmobilenetv2, two cameras and CPU detector on J5005 I have bad inference speed, going to over 300ms sometimes when eg. car headlights makes motion detector go crazy. I would really need to get GPU detector working.

[Jonoox]

I upgraded the host from Intel J5005 terminal to Intel N100 chinese box and everything works with frigate almost without any special configurations. Openvino GPU acceleration and 15-20ms inference speed with barely any CPU usage.

[ivanjx]

@Jonoox do you use hardware acceleration too?

[Jonoox]

Yes, vaapi preset. My frigate is running on proxmox, installed by tteck script.
vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()

[giuliodali]

Hi, I'm running Frigate 0.15-beta2 on an Intel J4125 as an add-on to Home Assistant OS

Core              2024.11.2
Supervisor        2024.11.2
Operating System  13.2
Frontend          20241106.2

with frigate config:

detectors:
  ov_0:
    type: openvino
    device: GPU
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

I get this in the logs too:

2024-11-20 22:40:51.631246947  Process detector:ov_0:
2024-11-20 22:40:51.631253833  Traceback (most recent call last):
2024-11-20 22:40:51.631255680    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-11-20 22:40:51.631262348      self.run()
2024-11-20 22:40:51.631264933    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2024-11-20 22:40:51.631266673      return run(*args, **kwargs)
2024-11-20 22:40:51.631313014    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-11-20 22:40:51.631314993      self._target(*self._args, **self._kwargs)
2024-11-20 22:40:51.631317433    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2024-11-20 22:40:51.631319102      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-11-20 22:40:51.631320869    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2024-11-20 22:40:51.631322325      self.detect_api = create_detector(detector_config)
2024-11-20 22:40:51.631363185    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-11-20 22:40:51.631364958      return api(detector_config)
2024-11-20 22:40:51.631387101    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 40, in __init__
2024-11-20 22:40:51.631389073      self.interpreter = self.ov_core.compile_model(
2024-11-20 22:40:51.631391326    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 543, in compile_model
2024-11-20 22:40:51.631392964      super().compile_model(model, device_name, {} if config is None else config),
2024-11-20 22:40:51.631394571  RuntimeError: Exception from src/inference/src/cpp/core.cpp:121:
2024-11-20 22:40:51.631395986  Exception from src/inference/src/dev/plugin.cpp:53:
2024-11-20 22:40:51.631397611  Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:185:
2024-11-20 22:40:51.631421102  [GPU] ProgramBuilder build failed!
2024-11-20 22:40:51.631423253  Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:429:
2024-11-20 22:40:51.631424646  [GPU] clWaitForEvents, error code: -14

I've found it discussed here and here on the intel boards, discussing Frigate.

The suggestions are:

• Ensure frigate is on the latest intel/compute-runtime 24.39.31294.12, which is newer than what was recently pinned
• From here setting i915.enable_hangcheck=0 i915.request_timeout_ms=100000 in grub as per docs.

Can these be tried?

FYI - @NickM-27

[NickM-27]

To be clear Celeron is not officially supported for openvino

[giuliodali]

According to this copy of the docs, as long as the GNA plugin is enabled then the Celeron J4125 would work?

(Config above fixed)

[NickM-27]

GNA does not support running object detection models

[luckydevil13]

try linux 6.6 kernel

[nuddler]

@giuliodali Do You managed it on GPU ?