Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Support]: High CPU usage for ffmpeg process #6461

Closed
poulti opened this issue May 10, 2023 · 20 comments
Closed

[Support]: High CPU usage for ffmpeg process #6461

poulti opened this issue May 10, 2023 · 20 comments

Comments

@poulti
Copy link

poulti commented May 10, 2023

Describe the problem you are having

CPU usage of the ffmpeg process is high, and slighlty higher (!) with hwaccel turned on. I'd like to optimise it as I can't scale down the resolution further on some cameras (G4s) and CPU usage is still high.

I'm running on a Raspberry Pi CM4 with 8GB RAM, and I've set GPU memory to 256MB.
I'm using the cameras named "Tapo2" and "G42" just to watch the stream in Home Assistant. No detection, no recording. And most of the time, little change there. The camera named "G41" is in the middle of the house, with lots of movements, and detection enabled (processed by the Coral USB).

Tests done (G41 and G42 have very similar CPU usage despite different configs) - see pictures below

  • (1) Tapo2 on 640x360 stream (lowest quality), no hwaccel --> ffpmeg CPU process around 10% (HAPPY)
  • (2) Tapo2 on 640x360 stream (lowest quality), with hwaccel --> ffpmeg CPU process around 14% (still HAPPY, but surprising)
  • (1) G41 on 1280x720 stream (lowest quality!), no hwaccel --> ffpmeg CPU process around 68% (SAD)
  • (2) G41 on 1280x720 stream (lowest quality!), with hwaccel --> ffpmeg CPU process around 74% (SAD)
  • (3) Tapo2 on 1920x1080 stream (high quality), no hwaccel --> ffpmeg CPU process around 77% (SAD)

What I don't understand:

  • Between Tapo2 low resolution and a G4 there are 4x more pixels (and twice more fps?) --> CPU usage is way higher (10% vs 68%)
  • Between a G4 and Tapo2 high resolution there are 2x more pixels --> CPU usage is slightly higher (68% vs 77%)
  • Why is hwaccel CPU usage for ffmpeg worse?
  • Is there any unwanted operations happening in ffmpeg (resizing?) I could deactivate?

Is my configuration incorrect in some ways? Should I add other arguments to ffmpeg input/output config? Any other suggestion?
(some links from #3860 are now dead, and couldn't find another idea to try from other "high CPU usage" issues)

Thanks a lot for the guidance

Version

0.12.0-DA3E197

Frigate config file (Test number 2: low res, with hwaccel)

# yaml-language-server: $schema=http://localhost:5555/api/config/schema
mqtt:
  host: mosquitto

## Hardware acceleration for raspberry 4 (64bit OS)
ffmpeg:
  hwaccel_args: preset-rpi-64-h264
# ffmpeg:
#   hwaccel_args:
#     - -c:v
#     - h264_v4l2m2m

cameras:

  camera_tapo2: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:xxx@xxx:554/stream2 # Low 640x360 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection in kids room
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 360

  camera_g41: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: true
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_g42: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection for FFA
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_iegeek1: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:554/2 # Low 640x352 stream
          roles:
            - detect
    motion:
      mask:
        #- 0,0,0,981,523,142,1212,105,1920,326,1920,0 # Mask for High quality stream
        - 0,0,0,352,139,32,344,31,640,260,640,0 # Mask for Low quality stream
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: True
      width: 640 # <---- update for your camera's resolution
      height: 352 # <---- update for your camera's resolution
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.8
    record:
      enabled: true
      retain:
        days: 0
        mode: motion
      events:
        retain:
          default: 2
          mode: active_objects
        pre_capture: 5
        post_capture: 5
        objects:
          - person
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 352

snapshots:
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: True


detectors:
  coral:
    type: edgetpu
    device: usb

Relevant log output

Nothing very relevant to me here...

2023-05-10 20:13:39.176710078  [INFO] Starting Frigate...
2023-05-10 20:13:43.706241317  [2023-05-10 20:13:43] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-05-10 20:13:43.847165359  [2023-05-10 20:13:43] peewee_migrate                 INFO    : Starting migrations
2023-05-10 20:13:43.917734907  [2023-05-10 20:13:43] peewee_migrate                 INFO    : There is nothing to migrate
2023-05-10 20:13:44.019192115  [2023-05-10 20:13:44] detector.coral                 INFO    : Starting detection process: 292
2023-05-10 20:13:44.029202701  [2023-05-10 20:13:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-05-10 20:13:44.036203392  [2023-05-10 20:13:44] frigate.app                    INFO    : Output process started: 294
2023-05-10 20:13:44.080914675  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_tapo2: 300
2023-05-10 20:13:44.117291725  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g41: 302
2023-05-10 20:13:44.138320408  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g42: 303
2023-05-10 20:13:44.182987895  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_iegeek1: 305
2023-05-10 20:13:44.215882201  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_tapo2: 307
2023-05-10 20:13:44.237082641  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g41: 308
2023-05-10 20:13:44.276377793  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g42: 312
2023-05-10 20:13:44.329073553  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_iegeek1: 317
2023-05-10 20:13:46.818909676  [2023-05-10 20:13:46] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

FFprobe output from your cameras

Tapo2:
  Metadata:
    title           : Session streamed by "TP-LINK RTSP Server"
    comment         : stream2
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 640x360, 15 fps, 28.58 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

G41:
  Metadata:
    title           : D021F99547EB_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

G42:
  Metadata:
    title           : D021F995415F_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

Frigate stats

See screen shots below

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

1x Tapo c200, 2x Ubiquiti G4, 1x ieGeek whatever

Any other information that may be helpful

  • Test (1) - low res, no hwaccel
    Screenshot 2023-05-09 215927_Tapo2_stream2_nohwaccel
  • Test (2) - low res, with hwaccel
    Screenshot 2023-05-09 215323_Tapo2_stream2_hwaccel
  • Test (3) - tapo high res, no hwaccel
    Screenshot 2023-05-10 213401_Tapo2_stream1_nohwaccel
@blakeblackshear
Copy link
Owner

Can you give this a try?

ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m

@poulti
Copy link
Author

poulti commented May 11, 2023

Thank you for the swift reply. It looks better for a hwaccel setting, ie at the level of the non hwaccel (at least it's not worse!) but not significantly better than non hwaccel.

  • Tapo Low res + NEW hwaccel settings
    Screenshot 2023-05-11 080003_Tapo2_lowres_NEWhwaccel
  • Tapo High res + NEW hwaccel settings
    Screenshot 2023-05-11 080414_Tapo2_highres_NEWhwaccel

@rovingclimber
Copy link

Hi!

Glad this has been raised actually - it's on my backlog to look at - I have 6 TP-link TAPO cameras (various models) which work really well with Frigate but definitely seeing higher CPU usage that I'd like (Intel J5040 / Coral PCIE). I've had to use the 1080p stream for detect on 3 of them because I'm looking specifically to detect our cats coming and going, and on the low-res streams a cat just don't give enough pixels :)

I've been using preset-intel-qsv-h264 so far, will try the setting above and see if it improves.

@blakeblackshear
Copy link
Owner

@rovingclimber this issue is specific to raspberry pi and it's hardware acceleration. I would suggest making your own issue as this issue isn't really relevant to your hardware.

@blacknell
Copy link
Contributor

I'm running into a similar problem. I have an RPi4 8GB with the follow CPU utilisations. Two cameras same model.

  • Acceleration off. 26% and 29%
  • Acceleration preset-rpi-64-h264 76% and 72%
  • Acceleration -c:v:1 and h264_v4l2m2m 29% and 26%

I'm also not getting any usage stats from GPUs (I am not running the addon - Frigate is standalone). GPU memory set to 1024MB

@NickM-27
Copy link
Sponsor Collaborator

The RPi does not support usage stats so that is expected

@blacknell
Copy link
Contributor

I should have left that point out as it's not my primary question. I'm most interested, and confused, about the CPU load given hardware acceleration settings.

@ferrarimarco
Copy link

ferrarimarco commented May 18, 2023

I'm experiencing the same issue. My setup similar (Raspberry Pi 4 8GB).

The CPU usage drops from about 45% to 13% after switching from preset-rpi-64-h264 to -c:v:1 h264_v4l2m2m with a single camera (2560x1920, resized to defaults for detection). Changing the configuration to avoid resizing makes the CPU load to jump to even higher values.

If I read the code correctly, that preset corresponds to:

"preset-rpi-64-h264": ["-c:v", "h264_v4l2m2m"],

According to the ffmpeg docs, stream_type[:additional_stream_specifier] is defined as:

stream_type is one of following: ’v’ or ’V’ for video, ’a’ for audio, ’s’ for subtitle, ’d’ for data, and ’t’ for attachments. ’v’ matches all video streams, ’V’ only matches video streams which are not attached pictures, video thumbnails or cover arts. If additional_stream_specifier is used, then it matches streams which both have this type and match the additional_stream_specifier. Otherwise, it matches all streams of the specified type.

So, if I get the v:1 selector correctly, it matches a specific stream (1), while v matches all video streams to set the -c h264_v4l2m2m option.

I'm not sure about the reason behind this behavior :(

My GPU memory is set to 76 MB, which is the max recommended for Raspberry Pi 4:

$ vcgencmd get_mem gpu
gpu=76M

PS: I just found #6385 that seems to be about this.

@poulti
Copy link
Author

poulti commented May 19, 2023

PS: I just found #6385 that seems to be about this.

Indeed, it looks like it would include the ":1" selector to all users of the rpi preset.

I'm not sure about the reason behind this behavior :(

That's a good question!
And the overall CPU usage remains high even when "doing nothing" with the stream. Obviously the app (ffmpeg in particular) is not "doing nothing" but I don't know if/what I could change to reduce the load.

Is there any other optimisation somebody think off?

@ZdenekM
Copy link

ZdenekM commented Jun 6, 2023

Can you give this a try?

ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m

In my case, the load went from 80 % to about 30 %. Nice!

@poulti
Copy link
Author

poulti commented Jun 12, 2023

Can you give this a try?

ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m

In my case, the load went from 80 % to about 30 %. Nice!

That's encouraging! Could you post a bit more details on your camera settings? (stream size, frigate config for those etc.)
Thanks

@blakeblackshear
Copy link
Owner

This change is included in the preset as of 0.12.1

@blacknell
Copy link
Contributor

Off topic perhaps but I wanted this change too.

Don't understand why the new docker build isnt pulled automatically when I have image: ghcr.io/blakeblackshear/frigate:stable in my docker compose

@blakeblackshear
Copy link
Owner

Did you run docker compose pull?

@blacknell
Copy link
Contributor

blacknell commented Jun 12, 2023 via email

@blakeblackshear
Copy link
Owner

Compose doesnt pull new images on startup

@Lockie85
Copy link

I believe I'm also having the same issue so thought I'd join in sharing as much info as I can to help.

Hardware
Raspberry Pi 4 (rpi4-64) 4GB RAM
Tapo C110
Tapo C200

Setup
Home Assistant 2023.6.2
Supervisor 2023.06.2
Operating System 10.3
Frontend 20230608.0 - latest

Frigate Version
v0.12.1-367d724

Frigate config file

mqtt:
  enabled: False

detectors: 
  coral:
    type: edgetpu 
    device: usb

go2rtc:  
  streams:  
    driveway:  
      - rtsp://user:password@192.168.5.99:554/stream1
  office:
      - rtsp://user:password@192.168.5.102:554/stream1

cameras:
  Driveway:
    ffmpeg:
      hwaccel_args: preset-rpi-64-h264
      inputs:
        - path: rtsp://user:password@192.168.5.99:554/stream1
          roles:
            - detect
            - record
    detect:
      enabled: True
      width: 1280
      height: 720
      fps: 5
    record: 
      enabled: True

  Garden:
    ffmpeg:
      hwaccel_args: preset-rpi-64-h264
      inputs:
        - path: rtsp://user:password@192.168.5.102:554/stream1
          roles:
            - detect
            - record
    detect:
      enabled: True
      width: 1280
      height: 720
      fps: 5
    record: 
      enabled: True

Frigate Log On Startup

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
2023-06-21 14:31:27.174186204  [INFO] Preparing go2rtc config...
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-06-21 14:31:27.208175951  [INFO] Starting NGINX...
2023-06-21 14:31:27.227518293  [INFO] Starting Frigate...
s6-rc: info: service legacy-services successfully started
2023-06-21 14:31:27.367359919  [INFO] Got IP address from supervisor: 192.168.5.4
2023-06-21 14:31:27.560599410  [INFO] Got WebRTC port from supervisor: 8555
2023-06-21 14:31:28.695611775  [INFO] Starting go2rtc...
2023-06-21 14:31:28.940308055  15:31:28.940 INF go2rtc version 1.2.0 linux/arm64
2023-06-21 14:31:28.941822029  15:31:28.941 INF [api] listen addr=:1984
2023-06-21 14:31:28.943361298  15:31:28.943 INF [rtsp] listen addr=:8554
2023-06-21 14:31:28.944765143  15:31:28.944 INF [srtp] listen addr=:8443
2023-06-21 14:31:28.945795958  15:31:28.945 INF [webrtc] listen addr=:8555
2023-06-21 14:31:32.815050709  [2023-06-21 15:31:32] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-06-21 14:31:32.894624073  [2023-06-21 15:31:32] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-06-21 14:31:32.903393120  [2023-06-21 15:31:32] peewee_migrate                 INFO    : Starting migrations
2023-06-21 14:31:32.920107953  [2023-06-21 15:31:32] peewee_migrate                 INFO    : There is nothing to migrate
2023-06-21 14:31:32.983297774  [2023-06-21 15:31:32] detector.coral                 INFO    : Starting detection process: 299
2023-06-21 14:31:32.989949934  [2023-06-21 15:31:32] frigate.app                    INFO    : Output process started: 301
2023-06-21 14:31:32.995724998  [2023-06-21 15:31:32] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-06-21 14:31:33.019675980  [2023-06-21 15:31:33] frigate.app                    INFO    : Camera processor started for Driveway: 305
2023-06-21 14:31:33.041041452  [2023-06-21 15:31:33] frigate.app                    INFO    : Camera processor started for Garden: 308
2023-06-21 14:31:33.061334573  [2023-06-21 15:31:33] frigate.app                    INFO    : Capture process started for Driveway: 310
2023-06-21 14:31:33.083712193  [2023-06-21 15:31:33] frigate.app                    INFO    : Capture process started for Garden: 311
2023-06-21 14:31:35.741484032  [2023-06-21 15:31:35] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-06-21 14:31:37.205268812  [INFO] Starting go2rtc healthcheck service...

System Info
image

Additional Notes
After only recently purchasing the Coral USB Accelerator, I'm unsure it's made any difference to my Raspberry Pi CPU usage by Frigate. Perhaps there is something wrong with my setup. Or perhaps it relates to this bug/issue?

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Jul 22, 2023
@poulti
Copy link
Author

poulti commented Jul 22, 2023

Any hope of using less CPU on raspberry, or should I upgrade the hardware?
Thanks

@github-actions github-actions bot removed the stale label Jul 23, 2023
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Aug 23, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants