[Detector Support]: Periodic Stuck Detection on AMD iGPU #19853
Unanswered
mreid-tt
asked this question in
Detector Support
Replies: 4 comments 8 replies
|
The periodic "Detection appears to be stuck. Restarting detection process..." message when using Frigate with an AMD iGPU and ONNX detector is a known behavior observed on AMD integrated graphics, especially when running ONNX models like YOLOv9-s(1)(2)(3). This can be triggered by the detector process hanging or failing to make progress, often due to compatibility or stability issues with the ROCm/ONNX stack on certain AMD hardware(2)(4)(3). Key points and suggestions:
If you continue seeing periodic stuck detection, this is likely due to incomplete support for your specific iGPU in the ROCm/ONNX stack. There is no official workaround in the documentation beyond the steps you've already taken(2)(4)(3). For further troubleshooting:
There are no documented configuration changes that guarantee a fix for this on AMD iGPUs at this time(2)(4)(3). 📚 Sources:
Have another question? Just tag @inkeep. |
0 replies
|
If your AMD CPU have Ryzen AI, you should really try Openvino (CPU) (don't GPU/Auto) and optional use YoloNAS. |
4 replies
|
Does Frigate have support for using the NPU on AMD CPUs? I didn't read anywhere on the Object Detectors page that this was a supported configuration. |
|
Frigate DOC in OpenVINO said "OpenVINO is supported on 6th Gen Intel platforms (Skylake) and newer. It will also run on AMD CPUs despite having no official support for it." |
|
No, NPUs are currently not supported for AMD |
|
Yep, thank you for information, but I'm just running OpenVINO CPU detector on AMD 7940HS. |
Update on the reported problemIt has been a while since my original report. I am now running TrueNAS Community 25.10.3 server with the same AMD Ryzen 7 PRO 8845HS and integrated Radeon 780M Graphics. We are using the As such, I still get from time to time: Version0.17.1-416a9b7 Frigate config fileversion: 0.17-0
mqtt:
enabled: false
detectors:
onnx:
type: onnx
auth:
failed_login_rate_limit: 1/second;5/minute;20/hour
trusted_proxies:
- 172.16.0.0/12
model:
path: /config/model_cache/yolov9-s.onnx
labelmap_path: /labelmap/coco-80.txt
width: 320
height: 320
input_tensor: nchw
input_dtype: float
model_type: yolo-generic
birdseye:
mode: motion
ffmpeg:
hwaccel_args: preset-vaapi
detect:
enabled: true
objects:
track:
- person
- car
notifications:
enabled: true
email: [user]@gmail.com
record:
enabled: true
motion:
days: 30
alerts:
retain:
days: 30
mode: active_objects
detections:
retain:
days: 30
snapshots:
enabled: true
retain:
default: 30
classification:
custom:
Front Gate:
enabled: true
name: Front Gate
threshold: 0.8
state_config:
cameras:
front_gate:
crop:
- 0.5153260514724064
- 0.003571463265869824
- 0.7841820354476002
- 0.36204610856612846
motion: true
go2rtc:
streams:
front_gate_1:
- rtsp://frigate:[password]@192.168.10.85:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
front_gate_2:
- rtsp://frigate:[password]@192.168.10.85:554/Streaming/Channels/102?transportmode=unicast&profile=Profile_2
cameras:
front_gate:
enabled: true
friendly_name: Front Gate
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/front_gate_1
input_args: preset-rtsp-restream
roles:
- record
- path: rtsp://127.0.0.1:8554/front_gate_2
input_args: preset-rtsp-restream
roles:
- detect
live:
streams:
Stream 1: front_gate_1
Stream 2: front_gate_2
motion:
mask:
- 0.04,0.07,0.631,0.068,0.631,0.126,0.04,0.125
- 0.743,0.868,0.976,0.868,0.976,0.923,0.743,0.923
camera_groups:
Reid-Home:
cameras: birdseye
icon: LuHouse
order: 1docker-compose file or Docker CLI command$ sudo cat /mnt/.ix-apps/app_configs/frigate/versions/1.3.7/templates/rendered/docker-compose.yaml | jq
{
"services": {
"frigate": {
"cap_add": [
"CHOWN",
"DAC_OVERRIDE",
"FOWNER",
"KILL",
"PERFMON",
"SETGID",
"SETUID"
],
"cap_drop": [
"ALL"
],
"deploy": {
"resources": {
"limits": {
"cpus": "2",
"memory": "4096M"
}
}
},
"devices": [
"/dev/dri:/dev/dri",
"/dev/kfd:/dev/kfd"
],
"environment": {
"HSA_OVERRIDE_GFX_VERSION": "11.0.3",
"NVIDIA_VISIBLE_DEVICES": "void",
"TZ": "America/Port_of_Spain",
"UMASK": "002",
"UMASK_SET": "002"
},
"group_add": [
44,
107,
568
],
"healthcheck": {
"interval": "30s",
"retries": 5,
"start_interval": "2s",
"start_period": "15s",
"test": [
"CMD",
"curl",
"--request",
"GET",
"--silent",
"--output",
"/dev/null",
"--show-error",
"--fail",
"http://127.0.0.1:5000/api"
],
"timeout": "5s"
},
"image": "ghcr.io/blakeblackshear/frigate:0.17.1-rocm",
"platform": "linux/amd64",
"ports": [
{
"mode": "ingress",
"protocol": "tcp",
"published": 30193,
"target": 8971
},
{
"mode": "ingress",
"protocol": "tcp",
"published": 8554,
"target": 8554
},
{
"mode": "ingress",
"protocol": "tcp",
"published": 8555,
"target": 8555
},
{
"mode": "ingress",
"protocol": "udp",
"published": 8555,
"target": 8555
}
],
"privileged": false,
"restart": "unless-stopped",
"security_opt": [
"no-new-privileges=true"
],
"shm_size": "512M",
"stdin_open": false,
"tmpfs": [
"/tmp/cache:size=1073741824"
],
"tty": false,
"user": "0:0",
"volumes": [
{
"bind": {
"create_host_path": false,
"propagation": "rprivate"
},
"read_only": false,
"source": "/mnt/SSDs/Applications/frigate/config",
"target": "/config",
"type": "bind"
},
{
"bind": {
"create_host_path": false,
"propagation": "rprivate"
},
"read_only": false,
"source": "/mnt/HDDs/Applications/frigate/media",
"target": "/media",
"type": "bind"
},
{
"read_only": false,
"target": "/tmp",
"type": "volume",
"volume": {
"nocopy": false
}
}
]
}
},
"volumes": {},
"x-action-required": false,
"x-notes": "# Frigate\n\n## Security\n\n**Read the following security precautions to ensure that you wish to continue using this application.**\n\n---\n\n### Container: [frigate]\n\n#### Running user/group(s)\n\n- User: root\n- Group: root\n- Supplementary Groups: apps\n\n#### Passing Host Files, Devices, or Sockets into the Container\n\n- AMD GPU Device (/dev/kfd) - (Read/Write)\n- DRI Device (/dev/dri) - (Read/Write)\n\n---\n\nDefault credentials are printed in the logs\nduring the first run of the application.\n\n## Bug Reports and Feature Requests\n\nIf you find a bug in this app or have an idea for a new feature, please file an issue at\nhttps://github.com/truenas/apps\n",
"x-portals": [
{
"host": "0.0.0.0",
"name": "Web UI",
"path": "/",
"port": 30193,
"scheme": "https"
}
]
}Relevant Frigate log outputInstall methodTrueNAS Apps Market Object DetectorOther (ROCm) Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpfulAfter the GPU hang it sometimes recovers and I see both CPU and GPU utilisation in the bottom left corner. Sometimes it doesn't and I only see CPU. I don't know if this means that detection is only happening with the CPU in this circumstance so I typically restart Frigate to get GPU stats back. Hardware Info: |
0 replies
|
A GPU hang is something that will most likely be fixed by an amd-firmware or driver upgrade. Frigate 0.18 does use newer ROCm so perhaps that will improve things but that's all we can do in any case |
4 replies
|
Thanks for the feedback. Beyond fixing the root cause, can you explain a bit more about why sometimes after a hang I see AMD GPU details when it restarts the service and sometimes I don't? When I only see CPU, does it mean that it is trying to run the YOLOv9-s model on the CPU? Or is it eventually running on the GPU and it just doesn't report correctly in the app? Why I ask is that it seems to continue to work after a hang and regardless of whether it correctly detects the AMD GPU details or not my overall CPU usage doesn't seem to look all that different. |
|
the GPU hang seems to do with the gpu stats tool. We see similar where it gets stuck showing 100% after running some models even when the process was shutdown. |
|
Hmm, so does this mean the app itself is still running correctly and just unable to report GPU statistics? In other words, when this happens, should I actually restart Frigate or just leave it alone? What’s confusing me is that at the end of the log section containing the errors, I still see: That suggests to me that the model is still loading successfully. If that’s the case, is Frigate continuing to use the GPU for detection, or does it silently fall back to CPU detection instead? That’s the part I’m unclear about. Also, is there any mechanism to have the stats tool retry automatically after a delay so it can recover on its own without requiring a manual restart? |
|
Yes seems that way. Currently no after an error it will stay stopped. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Describe the problem you are having
I’m running a TrueNAS-SCALE-24.10.2.4 server with an AMD Ryzen 7 PRO 8845HS and integrated Radeon 780M Graphics. Object detection is set up using the
rocmFrigate image with an ONNX YOLOv9-s model I built following the documented instructions.Every so often, I encounter the message:
Detection appears to be stuck. Restarting detection process...Version
0.16.0-c2f8de9
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Install method
TrueNAS Apps Market
Object Detector
Other (ROCm)
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
When I first set this up, I tried running the YOLOv9-m model. After leaving it overnight, the app became very sluggish, and I noticed extremely high GPU usage. The app didn’t crash, but the model was consuming excessive CPU/GPU resources until it became unresponsive. I stopped the container and switched to the YOLOv9-s model, and I haven’t seen that behavior since.
Hardware Info:
All reactions