[HW Accel Support]: ffmpeg high cpu caused by "-c:a aac" in record stream #11498
-
Describe the problem you are havingI am running frigate on an odroid M1S (rockchip 3566 chipset), and huge thanks to @MarcA711 for his efforts on hardware acceleration, it works fantastically well... however I was still battling high CPU on ffmpeg - but have now figured out the cause and mitigation. The recording stream is using -c:a:aac, which is apparently triggering a surprising amount of CPU to re-encode the (already AAC) audio stream. Changing this to -c:a:copy has a dramatic effect - Bringing cpu usage down from ~85% to ~18% (on a 1920x1080 20fps stream) I was surprised that ffmpeg apparently reencodes the audio stream here rather than transparently copying it from the incoming source, which matches the format. See the log output section for details. I'm happy to help troubleshoot and can reproduce easily if there is anything else I can test to make things more optimized, in the meantime I'm overriding the 'record' output args with my fix. Thank you for all your hard work, this project is great! Version0.13.2-69D9A261 (@MarcA711 built this, uses new rockchip hwaccel params and ffmpeg) Frigate config file# yaml-language-server: $schema=http://127.0.0.1:5000/api/config/schema.json
birdseye:
enabled: true
mode: motion
#restream: true
mqtt:
enabled: true
host: 192.168.11.5
cameras:
driveway:
enabled: true
ffmpeg:
inputs:
- path: rtsps://192.168.11.182:7441/sJKCVajaBKvzO1RS # Med Res
roles:
- detect
#- record
- path: rtsps://192.168.11.182:7441/LvsMuZNLIj1hAeXE # High Res
roles:
- record
#- detect
motion:
mask:
- 0,0,0,408,138,391,209,257,206,193,233,131,255,75,360,90,349,58,280,41,274,0
- 1280,0,1280,26,1088,21,1002,32,960,0
zones:
near_house:
coordinates: 0,720,1280,720,1280,461,1040,283,0,387
# Masks for high res detection
# motion:
# mask:
# - 751,0,676,192,846,208,803,314,558,323,491,832,0,866,0,0
# - 2688,0,2688,283,2313,280,2037,181,2039,0
# objects:
# filters:
# person:
# mask: 717,411,757,416,760,384,712,380
detect:
fps: 5
width: 1280
height: 720
wyzecam_test:
enabled: True
ffmpeg:
inputs:
- path: rtsp://192.168.11.211:8554/1080p?mp4 # High Res
roles:
- record
- detect
#- path: rtsp://192.168.11.211:8554/360p?mp4 # Low Res
#roles:
# - detect
motion:
mask:
- 1378,1080,1920,1080,1920,1007,1378,1002 # datetime stamp on wyze cams
detect:
fps: 5
width: 1920
height: 1080
objects:
filters:
bird:
min_score: 0.4
threshold: 0.5
wyzecam_indoor:
enabled: True
ffmpeg:
inputs:
- path: rtsp://192.168.11.168:8554/1080p?mp4 # High Res
roles:
- record
- detect
#- path: rtsp://192.168.11.168:8554/360p?mp4 # Low Res
# roles:
# - detect
motion:
mask:
- 1378,1080,1920,1080,1920,1007,1378,1002 # datetime stamp on wyze cams
detect:
fps: 5
width: 1920
height: 1080
detectors:
coral1:
type: edgetpu
device: pci:0
ffmpeg:
#global_args: "-v info -loglevel info -threads 2"
hwaccel_args: preset-rk-h264
output_args:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy
objects:
track:
- person
- dog
- car
- bicycle
- cat
- motorcycle
- bird
- bear
- sheep
- truck
snapshots:
enabled: True
record:
enabled: True
events:
retain:
default: 30
mode: motiondocker-compose file or Docker CLI commandfrigate:
container_name: frigate
restart: unless-stopped
image: docker.io/marca711/frigate:latest-rk
shm_size: "256mb"
environment:
- "PLUS_API_KEY=REDACTED"
volumes:
- ./frigate/config:/config
- /sdcard/frigate:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "5000:5000"
- "8554:8554" # rtsp
network_mode: host
privileged: true
# TODO: I don't think this is necessary when running privileged...
devices:
- /dev/apex_0:/dev/apex_0
- /dev/dri:/dev/dri
- /dev/dma_heap:/dev/dma_heap
- /dev/mali0:/dev/mali0
- /dev/rga:/dev/rga
- /dev/mpp_service:/dev/mpp_service
- /dev/iep:/dev/iep
- /dev/mpp-service:/dev/mpp-service
- /dev/vpu_service:/dev/vpu_service
- /dev/vpu-service:/dev/vpu-service
- /dev/hevc_service:/dev/hevc_service
- /dev/hevc-service:/dev/hevc-service
- /dev/rkvdec:/dev/rkvdec
- /dev/rkvenc:/dev/rkvenc
- /dev/vepu:/dev/vepu
- /dev/h265e:/dev/h265eRelevant log outputTesting within the docker container using 'docker exec -ti frigate /bin/bash:
# This runs at ~85% CPU
ffmpeg -loglevel trace -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent "FFmpeg Frigate/0.13.2-69d9a261" -avoid_negative_ts make_zero -fflags +discardcorrupt+genpts -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.11.211:8554/1080p?mp4 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac -f mp4 -f null -
Last message repeated 1 times
[h264 @ 0x5580a366c0] decode_slice_header error
[h264 @ 0x5580a366c0] no frame!
Input #0, rtsp, from 'rtsp://192.168.11.211:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 1716491616.152750, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
Metadata:
title : go2rtc/1.8.5
encoder : Lavf60.16.100
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (LC), 16000 Hz, stereo, fltp, 128 kb/s
Metadata:
encoder : Lavc60.31.102 aac
[aac @ 0x5580a7f050] Queue input is backward in time /s speed=N/A
[null @ 0x5580a54b20] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 62008 >= 60987
[aac @ 0x5580a7f050] Queue input is backward in time
[null @ 0x5580a54b20] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 89407 >= 88384
..... Those errors continue periodically .....
###############################################################################
# This runs at ~18% CPU
ffmpeg -loglevel info -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent "FFmpeg Frigate/0.13.2-69d9a261" -avoid_negative_ts make_zero -fflags +discardcorrupt+genpts -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.11.211:8554/1080p?mp4 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy -f mp4 -f null -
ffmpeg version 6b13265d70-20240215 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.0 (crosstool-NG 1.25.0.232_c175b21)
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-nonfree --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --disable-libdavs2 --enable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --disable-libvpx --enable-libwebp --enable-lv2 --disable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-rkmpp --enable-rkrga --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lstdc++ -lgomp' --extra-version=20240215
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
[h264 @ 0x5583e7b6c0] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x5583e7b6c0] decode_slice_header error
[h264 @ 0x5583e7b6c0] no frame!
[h264 @ 0x5583e7b6c0] non-existing PPS 0 referenced
Last message repeated 1 times
.... these continue for ~4 seconds ....
Last message repeated 1 times
[h264 @ 0x5583e7b6c0] decode_slice_header error
[h264 @ 0x5583e7b6c0] no frame!
Input #0, rtsp, from 'rtsp://192.168.11.211:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 1716491478.067500, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Output #0, null, to 'pipe:':
Metadata:
title : go2rtc/1.8.5
encoder : Lavf60.16.100
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
.... No errors like the prior command ....FFprobe output from your cameraInput #0, rtsp, from 'rtsp://192.168.11.168:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 20.08 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltpOperating systemOther Linux Install methodDocker Compose Network connectionWired Camera make and modelWyze Cam v3 (wz_mini_hacks) Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
this is exactly what would be expected. -c:a copy means to copy, otherwise -c:a aac is explicitly telling it to re-encode. The incoming audio stream can be aac but encoded to a different sample rate for example. Also there is on reason for you to use manual ffmpeg args. Finally, you are running a different ffmpeg build to the one that is used in mainline frigate, perhaps this version specifically has some issue with audio encoding. In my testing audio encoding in the main build is negligible. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick response! I wasn't aware there was an ffmpeg preset for audio copy, I'll start using that. I went back and looked at the ffmpeg trace log, you're right that it's reencoding, from aac-he to aac: FWIW I'm not exactly using a custom ffmpeg build, @MarcA711 added an updated one for hardware encoding support (in 15a6cc4), and I pulled the docker image he uploaded, then ran ffmpeg within the docker container to make sure I was in the same environment. My impression is that this will be in the next minor version build. I'll add the audio copy preset to the answer section, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
To summarize, aac was being re-encoded due to different profiles, the solution here is one of two things.
The camera was exporting an HE-AAC stream, which was being re-encoded by ffmpeg to an LC aac stream: |
Beta Was this translation helpful? Give feedback.
-
|
Hey, glad that everything works for you now :) I am still a bit surprised that re-encoding uses that much CPU. Sometimes loadavg is more precise. It would be interesting to see the output of One note: If you want you can also test the current dev version of frigate. The version itself should be considered unstable (so don't rely on it), but the rockchip specific part should work at this point. I am very thankful for any feedback! Docker image: ghcr.io/blakeblackshear/frigate:dev-058e9a3-rk |
Beta Was this translation helpful? Give feedback.
-
|
Since my job is troubleshooting production services, I like to be precise, so I set up some additional testing. I'll backport some of the findings here into the proposed answer so future people using wz_mini_hacks know that a simple change in the go2rtc encoding parameters will make a big change in aac encoding efficiency, presumably because ffmpeg is matching the input and output codec + profile and skipping re-encoding when they match Hope this all helps! Frist - wz_mini_hacks sets up a go2rtc instance in the wyze camera (ingenic t31), which gives the ability to tweak the encoding parameters at the source streams - here's what the default is for encoding: I left everything alone there, but duplicated the stream ("1080p_lc") and set the audio profile to "aac_low": With my two test streams set up, I shut down everything else on the odroid m1s and ran ffmpeg manually to get clean baseline runs. I'm using the 'timeout' and 'time' utilities in linux to run things for a set period and capture cpu stats. Summary:
first outputSecond output |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, so it really uses that much CPU to re-encode audio. Thank you for testing! |
Beta Was this translation helpful? Give feedback.
To summarize, aac was being re-encoded due to different profiles, the solution here is one of two things.
If you are able to change the source encoding (I'm using a wyze cam with wz_mini_hacks, so go2rtc encoding parameters are able to be changed), change the encoding profile to 'aac_low' and try to make sure the bitrate is 128k
This gives the most compatibility, as some clients cannot play back aac-he\
Set the frigate ffmpeg 'record' output arg to
preset-record-generic-audio-copyprofile to avoid re-encoding the audio track.This will work, but certain players (EG android chrome) cannot play the resulting stream
The camera was exporting an HE-AAC stream, which was being re-encoded …