Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Support]: hwaccel stopped working on raspberry pi after 5.15.61 kernel update #3780

Closed
kylehendricks opened this issue Sep 7, 2022 · 172 comments

Comments

@kylehendricks
Copy link

kylehendricks commented Sep 7, 2022

Describe the problem you are having

Everything was working fine with my frigate setup. I've been using the 0.11 betas/rc's since the beginning. All of the sudden one of my cameras stopped working in frigate completely. Blue Iris was still working fine with it. VLC opened the configured URL no problem. I've issues once in a while in the past where I'd see events stop happening so I'd just bounce frigate and all would be well. This time, bouncing frigate and the rebooting the camera hasn't helped.

The really strange thing is that I can't think of anything that has changed. IIRC, the camera was working fine with 0.11RC2 for at least a few days before this randomly started happening. No matter what I tried I just get repeating No frames received from driveway in 20 seconds errors. I do have another camera that is the same brand but a little bit older model that continued to work fine. I removed it from the config since the issue with the one camera still occurs.

Version

0.11.0-c461c9e

Frigate config file

mqtt:
  host: mqtt.home
  user: 'XXXX'
  password: 'XXXXXXX'

database:
  path: /db/frigate.db

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

record:
  enabled: True
  events:
    retain:
      default: 30

snapshots:
  enabled: True
  retain:
    default: 30

rtmp:
  enabled: False

cameras:
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXX:XXXXXX@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - rtmp
            - record
        - path: rtsp://XXXXXX:XXXXXXX@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1230,441,1252,310,1259,120,1257,0,1280,0,1280,720,1130,720
        - 396,116,649,67,824,57,1002,70,1134,94,1259,122,1257,0,949,0,819,0,626,0,0,0,0,260
    zones:
      driveway_entry:
        coordinates: 819,248,639,209,1152,99,1202,124
      driveway_main:
        coordinates: 1095,720,332,720,244,560,174,308,425,217,476,233,528,225,630,214,821,266,867,305,1207,433
    objects:
      track:
        - person
        - car
        - dog

Relevant log output

frigate  | [2022-09-07 02:01:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.                                                       
frigate  | [2022-09-07 02:01:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...                                         
frigate  | [2022-09-07 02:01:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.                                                          
frigate  | [2022-09-07 02:01:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.                                        
frigate  | [2022-09-07 02:01:45] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:53484]                                
frigate  | [2022-09-07 02:01:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...                                          
frigate  | [2022-09-07 02:01:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...                                                                   
frigate  | [2022-09-07 02:01:50] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51066]                                   
frigate  | [2022-09-07 02:02:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:02:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:02:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:02:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:02:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:02:37] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:56424]
frigate  | [2022-09-07 02:02:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:02:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:02:50] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51066]
frigate  | [2022-09-07 02:02:55] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45176]
frigate  | [2022-09-07 02:03:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:03:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:03:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:03:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:03:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:03:55] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45176]
frigate  | [2022-09-07 02:04:00] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51724]
frigate  | [2022-09-07 02:04:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:04:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:04:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:04:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:04:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:05:00] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51724]
frigate  | [2022-09-07 02:05:05] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58226]
frigate  | [2022-09-07 02:05:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:05:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:05:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:05:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:05:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:05:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:05:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...

FFprobe output from your camera

root@edcaa9ad0fcf:/opt/frigate# ffprobe "rtsp://xxxxx:xxxxx@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --disable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --disable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://xxxxx:xxxxx@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt470bg/bt470bg/bt709, progressive), 2688x1520, 10 fps, 10 tbr, 90k tbn
  Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s

Frigate stats

{
  "detection_fps": 0,
  "detectors": {
    "coral": {
      "detection_start": 0,
      "inference_speed": 10,
      "pid": 215
    }
  },
  "driveway": {
    "camera_fps": 0,
    "capture_pid": 227,
    "detection_fps": 0,
    "pid": 221,
    "process_fps": 0,
    "skipped_fps": 0
  },
  "service": {
    "latest_version": "0.10.1",
    "storage": {
      "/dev/shm": {
        "free": 133.9,
        "mount_type": "tmpfs",
        "total": 134.2,
        "used": 0.3
      },
      "/media/frigate/clips": {
        "free": 9131015.5,
        "mount_type": "cifs",
        "total": 19859668.5,
        "used": 10728652.9
      },
      "/media/frigate/recordings": {
        "free": 9131015.5,
        "mount_type": "cifs",
        "total": 19859668.5,
        "used": 10728652.9
      },
      "/tmp/cache": {
        "free": 954,
        "mount_type": "tmpfs",
        "total": 1000,
        "used": 46
      }
    },
    "temperatures": {},
    "uptime": 1041,
    "version": "0.11.0-c461c9e"
  }
}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Dahua IPC-T5442T-ZE

Any other information that may be helpful

No response

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Sep 7, 2022

It seems you're running on the RPi and using hwaccel. Have you tried it without hwaccel?

@kylehendricks
Copy link
Author

That was it!

I'm still thoroughly confused as to why this suddenly started happening. I even tried beta7/rc1 and they are no longer working either. My other 1080p Dahua also still works with hwaccel on...

Is this a known thing that hwaccel sometimes just doesn't work on a rpi4?

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Sep 7, 2022

What is your GPU memory set to? Odds are you have another program using too much memory so the Pis GPU is starving and unable to process the frames.

@kylehendricks
Copy link
Author

My gpu_mem is set to 256 according to my /boot/config.txt

This pi4 is solely being used for frigate.

As I was typing this, one thing I remember did change recently is i ran a apt full-upgrade on the pi recently. That has to be related to this sudden issue.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Sep 7, 2022

Maybe try increasing it to 512mb

@kylehendricks
Copy link
Author

I was thinking the max was 256 for some reason. Tried bumping to 512 and rebooting, exact same behavior when hwaccel is on.

I just looked at my /var/log/apt/history.log and saw my most recent apt upgrade was right around when the issue started happening. Here's what was updated:

raspberrypi-bootloader:arm64 (1:1.20220811-1, 1:1.20220830-1)
libcamera-apps-lite:arm64 (0~git20220707+35266e8-1, 0~git20220830+1bf0cca-1)
libcamera0:arm64 (0~git20220705+f30ad033-1, 0~git20220826+3fad116f-1)
raspberrypi-kernel:arm64 (1:1.20220811-1, 1:1.20220830-1)
linux-libc-dev:arm64 (1:1.20220811-1, 1:1.20220830-1)

Could be the kernel or bootloader update?

@kylehendricks
Copy link
Author

I don't know much about ffmpeg and hardware acceleration but I searched the raspberry pi kernel issues for the kernel version I just updated to (5.15.61-v8+) and found a couple recent issues that may be related:
raspberrypi/linux#5150
raspberrypi/linux#5166

@kylehendricks
Copy link
Author

ffprobe output for a camera that's currently working with hwaccel:

ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --disable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --disable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://xxxxx:xxxxxx@south-garage.cam.home:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.200000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 10 fps, 10 tbr, 90k tbn

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Sep 7, 2022

We've seen other users complain that the latest RPi kernel broke hwaccel as well

@kylehendricks kylehendricks changed the title [Support]: One camera suddenly stopped working with repeating "No frames received from driveway in 20 seconds" errors [Support]: hwaccel stopped working on raspberry pi after 5.15.61 kernel update Sep 7, 2022
@LaurenceGough
Copy link

LaurenceGough commented Sep 8, 2022

Yeah same issue here sadly. :( Turning off hw Accel works for now but the poor thing is deffo working harder even with just two cameras setup.

It happens with rc1 and rc2 for me. I'm using two different hikvision cameras sadly neither work. H264.

If I find a better fix I'll be sure to share

@idontcare99999
Copy link

idontcare99999 commented Sep 14, 2022

Same here. Fresh install of Raspberry Pi OS Lite 2022-09-06 with kernel 5.15.61-v8+ on a Raspberry Pi 4 Model B Rev 1.2 running docker and only green screen when using hwaccel_args:
- -c:v
- h264_v4l2m2m

Just tried crzynik/frigate:ffmpeg-update and same result.

Seems the issue has been identified: raspberrypi/linux#5150

@idontcare99999
Copy link

and the fix is in...
raspberrypi/linux#5150 (comment)

@joe248
Copy link

joe248 commented Sep 24, 2022

I'm experiencing this issue as well since updating the PI kernel and trying to figure out a solution. My understanding is that:

  1. The kernel update broke things, but the issue is with FFmpeg
  2. The fix is in for the 4.3.4 version of FFmpeg packaged with the PI, but hasn't been released yet, though this won't solve the issue because Frigate uses a different version of FFmpeg inside the Docker container
  3. Frigate is using a BtbN build of FFmpeg 5.1 which appears to be auto-built from the official FFmpeg repo

After looking at the fix that @idontcare99999 mentioned, I checked the official FFmpeg repo and it doesn't appear to contain the fix. Has anyone else managed to get hwaccel working again other than by downgrading to the previous kernel (which I'm not opposed to but have never done before)?

@LaurenceGough
Copy link

LaurenceGough commented Sep 24, 2022

I'm experiencing this issue as well since updating the PI kernel and trying to figure out a solution. My understanding is that:

  1. The kernel update broke things, but the issue is with FFmpeg
  2. The fix is in for the 4.3.4 version of FFmpeg packaged with the PI, but hasn't been released yet, though this won't solve the issue because Frigate uses a different version of FFmpeg inside the Docker container
  3. Frigate is using a BtbN build of FFmpeg 5.1 which appears to be auto-built from the official FFmpeg repo

After looking at the fix that @idontcare99999 mentioned, I checked the official FFmpeg repo and it doesn't appear to contain the fix. Has anyone else managed to get hwaccel working again other than by downgrading to the previous kernel (which I'm not opposed to but have never done before)?

Make sure you do a disk clone or otherwise full backup before messing with the kernel to save yourself many hours of work...

Yeah, I'm a complete Linux newbie so I most likely did something very wrong, It'd be nice to get HWaccel working again but I do like having Home assistant and Frigate running ;).

If anyone finds a way to downgrade the kernel please let me know. Thank you. Never again will I update without a full, tested backup.

@joe248
Copy link

joe248 commented Oct 3, 2022

I used the following process to downgrade the kernel from 5.15.61-v8+ to 5.15.32-v8+ and hwaccel is working again:

  1. Check /var/log/apt to find last kernel date version before upgrade. For me it was 1.20220331-1.
  2. Check https://github.com/raspberrypi/rpi-firmware/commits/master to find commit hash for that date (a54fe46c85fd4a2155f2282454bee3c2a3d5b5eb)
  3. Power down the PI, put the SD card into a computer, and create a backup image. For Mac OS I used: sudo dd bs=4m if=/dev/disk2 of=raspbian.img
  4. Put the SD card back into the PI and boot. Run sudo rpi-update a54fe46c85fd4a2155f2282454bee3c2a3d5b5eb
  5. Reboot

@jherby2k
Copy link

jherby2k commented Oct 3, 2022

Seems to affect Home Assistant OS (HASSOS) also (on Pi 4). Or at least this seems to be what i'm experiencing - new install, hwaccel has never worked.

@LaurenceGough
Copy link

LaurenceGough commented Oct 16, 2022

I am not sure if I should downgrade the kernel using Joe's instructions as kindly posted above or try to build the Pi specific FFMPEG build here in a test branch:

raspberrypi/linux#5150 (comment)

I am not sure how long we would have to wait for the next Pi ffmpeg update. Has anyone here tried to build this specific branch version? As of today hwaccel for Frigate on the Pi 4 fully updated still doesn't work unfortunately.

For now I am making do without hwaccel but the Pi is certainly struggling even with just two cameras I'd imagine this would be a nightmare for people with more.

I am not seeing too many people reporting this issue, I take it many are not updating their Pi's? :o

I might give the sudo rpi-update a54fe46c85fd4a2155f2282454bee3c2a3d5b5eb a go after making a full backup.

@idontcare99999
Copy link

I'm hoping I can wait it out...

I have built the fixed ffmpeg inside a few different docker containers but could never get it to work within the frigate container.

@jherby2k
Copy link

I'm surprised more people don't report this issue as well - anyone running HA on a Raspberry Pi via HA OS (So the most popular install type) would be experiencing this AFAICT.

@LaurenceGough
Copy link

The same as well for Raspberry Pi OS and also DietPi OS too.

@sparkydave1981

This comment was marked as off-topic.

@NickM-27
Copy link
Sponsor Collaborator

@sparkydave1981 make your own issue, that is unrelated to this one if it is not an RPi

@hackneyb
Copy link

hackneyb commented Jan 7, 2023

@blakeblackshear are you able to summarize what will resolve this issue, given your comment in the encoding issue that I linked to above yesterday, and you commented on as not related, a couple hours ago? Is https://trac.ffmpeg.org/ticket/10060 relevant to solving the issue we're experiencing and is it worded correctly? Thanks in advance

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Jan 7, 2023

A fix is already in progress #4955

@amzaldua
Copy link

amzaldua commented Jan 30, 2023

Hello,
Any new news about this matter?

@LaurenceGough
Copy link

It's fixed with the latest 0.12 beta builds :)

@erahhal
Copy link

erahhal commented Mar 8, 2023

Apologies for the confusion - I've read through this thread and various associated links and tickets here, and am unclear on the conclusion. Does 0.12 beta include a version of ffmpeg that works with newer kernels? Or do we still need to roll back the kernel? I've got 0.12 beta installed and am seeing the same issue (HAOS on a Raspi 4). Perhaps I'm not configuring things properly.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Mar 8, 2023

Apologies for the confusion - I've read through this thread and various associated links and tickets here, and am unclear on the conclusion. Does 0.12 beta include a version of ffmpeg that works with newer kernels? Or do we still need to roll back the kernel? I've got 0.12 beta installed and am seeing the same issue (HAOS on a Raspi 4). Perhaps I'm not configuring things properly.

This is fixed in 0.12. If you're having issues you should make a separate issue.

@erahhal
Copy link

erahhal commented Mar 9, 2023

Thanks, I've opened issue #5682

@SpartanTech
Copy link

Does this issue still exist in Frigate 11.1? My RPI4 kernel is 6.19-v8+, default for the Raspberry PI OS Install currently

@NickM-27
Copy link
Sponsor Collaborator

Does this issue still exist in Frigate 11.1? My RPI4 kernel is 6.19-v8+, default for the Raspberry PI OS Install currently

Yes, 0.11 does not contain a fix

@SpartanTech
Copy link

SpartanTech commented Mar 24, 2023

thank you for the quick reply! ive been racking my brain for 2 weeks off and on trying to get frigate to work. the beta has a ton of ffmpeg errors on my new install, the non beta has no errors but cant record / green snapshots. this helps me try something else :)

@NickM-27
Copy link
Sponsor Collaborator

thank you for the quick reply! ive been racking my brain for 2 weeks off and on trying to get frigate to work. the beta has a ton of ffmpeg errors on my new install, the non beta has no errors but cant record / green snapshots. this helps me try something else :)

I'd suggest making a new support issue then, likely a configuration issue on the beta

@rjpearce
Copy link

rjpearce commented Apr 8, 2023

rpi-update 109147b5dd3cdb736173be079efc03950c1d20b3 just a word of warning for the next person. This was destructive for me, it made my RPi 4B un-bootable from SSD. I would live without hardware acceleration if I had known the massive pain this would cause.

@joe248
Copy link

joe248 commented Jul 31, 2023

Checking in on this issue. I just installed Frigate 0.12.1 and hwaccel does not appear to be working (or at least not working as well as it used to). CPU usage for my cameras went from ~4.5% each to ~9% each. I am still running the older 5.15.32-v8+ kernel and wondering if I need to update to the latest to fix this.

I see some conflicting reports about whether or not this fix is working. Can anyone confirm that they are running Frigate 0.12.1 on the latest PI kernel, and that their CPU usage is similar to what it was when running Frigate 0.11 and the older kernel?

@joe248
Copy link

joe248 commented Aug 12, 2023

I just upgraded my kernel from 5.15.32-v8+ to 6.1.21-v8+. Interestingly, after doing that, Frigate 11.1 still runs fine with no CPU usage increase. So they must have changed something in the kernel between 5.15.61-v8+ and 6.1.21-v8+ that fixed the original issue.

However, after installing Frigate 12.1, my ffmpeg CPU usage still doubles. Running 11.1, my usage is ~4.5% per camera. Running 12.1, usage is:

  • ~9% per camera using hwaccel_args: preset-rpi-64-h264
  • ~9% per camera if I comment out #hwaccel_args:
  • 16-20% per camera if I use the args from 11.1 (hwaccel_args: -c:v h264_v4l2m2m)

It seems like whatever ffmpeg build Frigate 12 is using doesn't get along well with my PI.

@NickM-27
Copy link
Sponsor Collaborator

It seems like whatever ffmpeg build Frigate 12 is using doesn't get along well with my PI.

Frigate 0.12.x uses a version of ffmpeg built specifically for the RPi from the rpi repo

@joe248
Copy link

joe248 commented Aug 12, 2023

I did some testing by logging into the docker containers and running the same ffmpeg commands that Frigate is running.

Frigate 0.11.1 (ffmpeg version n5.1-2-g915ef932a3-20220731 ) :

Output #0, segment, to '/tmp/cache/Doorbell-%Y%m%d%H%M%S.mp4':
  Metadata:
    title           : Media Server
    encoder         : Lavf59.27.100
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 720x576, q=2-31, 15 fps, 15 tbr, 15360 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:0 -> #1:0 (h264 (h264_v4l2m2m) -> rawvideo (native))
Output #1, rawvideo, to 'pipe:':
  Metadata:
    title           : Media Server
    encoder         : Lavf59.27.100
  Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 720x576, q=2-31, 24883 kb/s, 5 fps, 5 tbn
    Metadata:
      encoder         : Lavc59.37.100 rawvideo

Frigate 0.12.1 (ffmpeg version 4.3.6-0+deb11u1+rpt1):

Output #0, segment, to '/tmp/cache/Doorbell-%Y%m%d%H%M%S.mp4':
  Metadata:
    title           : Media Server
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 720x576, q=2-31, 15 fps, 15 tbr, 15360 tbn, 15 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))
Output #1, rawvideo, to 'pipe:':
  Metadata:
    title           : Media Server
    encoder         : Lavf58.45.100
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x576, q=2-31, 24883 kb/s, 5 fps, 5 tbn, 5 tbc
    Metadata:
      encoder         : Lavc58.91.100 rawvideo

Notice the differences on the Stream #0:0 -> #1:0 lines. I assume that h264_v4l2m2m means it's using the hardware decoder. I'm not sure what native means. If I change the Frigate 0.12.1 command from -c:v:1 to -c:v, the stream changes to h264_v4l2m2m, however, CPU usage is higher (~15%). I'm not sure what to make of this, but I get the best performance by far with -c:v / Frigate 0.11.1 / ffmpeg 5.1.

What was the reasoning for using the older ffmpeg version? Was it solely due to it being a special build for RPi?

@NickM-27
Copy link
Sponsor Collaborator

I'm not sure what to make of this, but I get the best performance by far with -c:v / Frigate 0.11.1 / ffmpeg 5.1.

Then you can use the newer version of ffmpeg inside frigate 0.12, after all Frigate supports custom ffmpeg builds

What was the reasoning for using the older ffmpeg version?

The fact that hwaccel did not work at all for the vast majority of users, hence this issue existing and having ~150 comments

@joe248
Copy link

joe248 commented Aug 12, 2023

If I remove -c:v:1 h264_v4l2m2m (on Frigate 0.12.1) I get the same stream mapping as when I leave it in: Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native)). The CPU usage is also the same. This leads me to believe that preset-rpi-64-h264 (-c:v:1 h264_v4l2m2m) is not actually doing anything.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 12, 2023

This leads me to believe that preset-rpi-64-h264 (-c:v:1 h264_v4l2m2m) is not actually doing anything.

maybe in your case, many other users have found it to greatly reduce CPU usage. Of course, it depends on the camera pixel format, your host driver / kernel, etc.

#6053 (reply in thread)

The RPi decoder in general is not very robust

@joe248
Copy link

joe248 commented Aug 12, 2023

Then you can use the newer version of ffmpeg inside frigate 0.12, after all Frigate supports custom ffmpeg builds

I will try this next. I'm just attempting to help out the project and suggesting that maybe it's time to revisit ffmpeg 5.1 now that it appears something has changed in the kernel that fixed the original issue (at least for me). Hopefully others can run some tests as well to confirm.

maybe in your case, many other users have found it to greatly reduce CPU usage. Of course, it depends on the camera pixel format, your host driver / kernel, etc.

Sure - I'm not suggesting that no one should use those parameters. But the preset specifically for the PI (preset-rpi-64-h264) is doing nothing for me for any cameras (Amcrest, TP Link, Wyze) and I'd be interested to know if that's the case for others with PIs.

@hstr0100
Copy link

Then you can use the newer version of ffmpeg inside frigate 0.12, after all Frigate supports custom ffmpeg builds

I will try this next. I'm just attempting to help out the project and suggesting that maybe it's time to revisit ffmpeg 5.1 now that it appears something has changed in the kernel that fixed the original issue (at least for me). Hopefully others can run some tests as well to confirm.

maybe in your case, many other users have found it to greatly reduce CPU usage. Of course, it depends on the camera pixel format, your host driver / kernel, etc.

Sure - I'm not suggesting that no one should use those parameters. But the preset specifically for the PI (preset-rpi-64-h264) is doing nothing for me for any cameras (Amcrest, TP Link, Wyze) and I'd be interested to know if that's the case for others with PIs.

Can confirm, same here, no change
Also I've been all day trying to configure a udp rtsp h265-pcm camera to record in h264 aac and no matter what scenario, when I get something that kinda works it absolutely pegs the cpu and the video gets insanely choppy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests