[HW Accel Support]: High CPU and zero iGPU usage with hardware acceleration

[albertomercurio]

Describe the problem you are having

I have two cameras that stream h264 rtsp. My system has an Intel N95, and I want to use the hardware acceleration. In my docker compose file, I also added the /dev/dri/card0 device, because when I run intel_gpu_top this is my output

intel-gpu-top: 8086:46d2 @ /dev/dri/card0 -  455/ 805 MHz;   0% RC6;      105 irqs/s

         ENGINES     BUSY                                                                                                                             MI_SEMA MI_WAIT
       Render/3D    0.69% |▉                                                                                                                        |      0%      0%
         Blitter    0.00% |                                                                                                                         |      0%      0%
           Video    0.66% |▉                                                                                                                        |      0%      0%
    VideoEnhance    0.00% |                                                                                                                         |      0%      0%

But maybe it is not important.

Although I set the vaapi hardware acceleration, I see that the cpu usage is very high (70% or higher), while the iGPU usage is almost zero (you can see in the example I showed above).

I tried to use also the QuickSync acceleration, but it doesn't work. I get and error and I can't see anything.

Version

0.12.1-367D724

Frigate config file

mqtt:
  enabled: false # I don't use Home Assistant automations for the moment

database:
  # The path to store the SQLite DB (default: shown below)
  path: /db/frigate.db

detectors:
  cpu:
    type: cpu
    num_threads: 4

ffmpeg:
  hwaccel_args: preset-vaapi
  # hwaccel_args: preset-intel-qsv-h264

go2rtc:
  streams:
    garage:
      - rtsp://192.168.1.27:8554/mainstream
    garage_sub:
      - rtsp://192.168.1.27:8554/substream
    cucina:
      - rtsp://192.168.1.24:8554/mainstream
    cucina_sub:
      - rtsp://192.168.1.24:8554/substream


objects:
  track:
    - person
  filters:
    person:
      threshold: 0.7


cameras:
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage_sub
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/garage
          roles:
            - record
    detect:
      enabled: true
      width: 720
      height: 480
    record:
      enabled: true
  cucina:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cucina_sub
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/cucina
          roles:
            - record
    detect:
      enabled: true
      width: 720
      height: 480
    record:
      enabled: true

docker-compose file or Docker CLI command

frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
    networks:
      - casanetwork
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      # - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/casa/docker-compose/frigate/config/config.yml:/config/config.yml
      - /home/casa/frigate/media:/media/frigate # Not in the docker-compose folder because it might be too big
      - /home/casa/docker-compose/frigate/db:/db
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

Relevant log output

2023-12-09 12:31:25.006702637  [INFO] Starting Frigate...
2023-12-09 12:31:25.931149409  [2023-12-09 12:31:25] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-12-09 12:31:25.948883481  [2023-12-09 12:31:25] peewee_migrate                 INFO    : Starting migrations
2023-12-09 12:31:25.952658066  [2023-12-09 12:31:25] peewee_migrate                 INFO    : There is nothing to migrate
2023-12-09 12:31:25.965941836  [2023-12-09 12:31:25] detector.cpu                   INFO    : Starting detection process: 289
2023-12-09 12:31:25.968958572  [2023-12-09 12:31:25] frigate.app                    INFO    : Output process started: 291
2023-12-09 12:31:25.969169287  [2023-12-09 12:31:25] frigate.detectors              WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
2023-12-09 12:31:25.976120795  [2023-12-09 12:31:25] frigate.app                    INFO    : Camera processor started for garage: 294
2023-12-09 12:31:25.982416023  [2023-12-09 12:31:25] frigate.app                    INFO    : Camera processor started for cucina: 296
2023-12-09 12:31:25.988026156  [2023-12-09 12:31:25] frigate.app                    INFO    : Capture process started for garage: 297
2023-12-09 12:31:25.994569349  [2023-12-09 12:31:25] frigate.app                    INFO    : Capture process started for cucina: 301

FFprobe output from your camera

ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://192.168.1.24:8554/substream':
  Metadata:
    title           : 
    comment         : substream
  Duration: N/A, start: 0.040000, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuvj420p(pc, progressive), 640x360, 25 fps, 25 tbr, 90k tbn, 180k tbc

Operating system

Other

Install method

Docker Compose

Network connection

Wired

Camera make and model

Hacked Xiaomi 360

Any other information that may be helpful

My system has Ubuntu 22.04

The Hacked firmware of my camera is available at this link

https://github.com/telmomarques/xiaomi-360-1080p-hacks

[NickM-27]

I would suggest trying the i965 driver

[albertomercurio]

I added the following in my docker compose yaml

    environment:
      LIBVA_DRIVER_NAME: i965

But now I can no longer see my camera output, with the following errors (only a part of):

2023-12-09 14:20:35.298052091  [2023-12-09 14:20:35] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-12-09 14:20:35.298058345  [2023-12-09 14:20:35] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-12-09 14:20:35.812959533  [2023-12-09 14:20:35] frigate.video                  ERROR   : cucina: Unable to read frames from ffmpeg process.
2023-12-09 14:20:35.813198993  [2023-12-09 14:20:35] frigate.video                  ERROR   : cucina: ffmpeg process is not running. exiting capture thread...
2023-12-09 14:20:49.645083074  [2023-12-09 14:20:49] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-12-09 14:20:49.645174608  [2023-12-09 14:20:49] frigate.record                 ERROR   : list index out of range
2023-12-09 14:20:54.257252367  [2023-12-09 14:20:54] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-12-09 14:20:54.257640988  [2023-12-09 14:20:54] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-12-09 14:20:54.257961477  [2023-12-09 14:20:54] ffmpeg.garage.detect           ERROR   : [AVHWDeviceContext @ 0x559bf34e8cc0] libva: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
2023-12-09 14:20:54.265182265  [2023-12-09 14:20:54] ffmpeg.garage.detect           ERROR   : [AVHWDeviceContext @ 0x559bf34e8cc0] Failed to initialise VAAPI connection: -1 (unknown libva error).
2023-12-09 14:20:54.265658960  [2023-12-09 14:20:54] ffmpeg.garage.detect           ERROR   : Device creation failed: -5.
2023-12-09 14:20:54.265827652  [2023-12-09 14:20:54] ffmpeg.garage.detect           ERROR   : [h264 @ 0x559bf34b0940] No device available for decoder: device type vaapi needed for codec h264.
2023-12-09 14:20:54.265994644  [2023-12-09 14:20:54] ffmpeg.garage.detect           ERROR   : Device setup failed for decoder on input stream #0:0 : Input/output error

[NickM-27]

Okay in this case let's remove that and run vainfo on the host and compare that to frigate

[albertomercurio]

On my host I get:

error: can't connect to X server!
error: failed to initialize display

On Frigate container I get:

error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointEncSlice
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSlice
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointEncSlice
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

[NickM-27]

The driver would be expected on the host. It's possible it is very out of date and updating it would help

[albertomercurio]

Ok I managed to obtain the output, I didn't run it with sudo. Now the output on the host is:

libva info: VA-API version 1.14.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

[NickM-27]

We can see the driver is a version behind what is in frigate, might help to update the driver

[albertomercurio]

How can I update it? I didn't find anything on the internet.

[albertomercurio]

sudo apt install intel-media-va-driver says that it is already at the latest version

intel-media-va-driver è già alla versione più recente (22.3.1+dfsg1-1ubuntu2).

[albertomercurio]

I'm trying to figure out why the quiksync doesn't work for my Frigate container, while it does work in my Jellyfin container.

I run the following code on both containers:

ffmpeg -hwaccel qsv -c:v h264_qsv -i rtsp://192.168.1.24:8554/mainstream -vf hwdownload,format=nv12 -pix_fmt yuv420p prova.mp4

But the Frigate container returned the following error:

ffmpeg version n5.1-2-g915ef932a3-20220731 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".
Input #0, rtsp, from 'rtsp://192.168.1.24:8554/mainstream':
  Metadata:
    title           : 
    comment         : mainstream
  Duration: N/A, start: 0.040000, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuvj420p(pc, progressive), 1920x1088, 25 fps, 25 tbr, 90k tbn
[AVHWDeviceContext @ 0x555eef117540] Error setting child device handle: -17
Segmentation fault (core dumped)

And this is the vainfo output of my Jellyfin container:

Trying display: drm
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_20
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.20.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.4.0 (05e4d0d)
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointEncSlice
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSlice
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointEncSlice
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

[albertomercurio]

Speaking of vaapi, I noticed that running the command below in my Frigate container, I can see a small amount of GPU use (I changed the substream with the mainstream to see better the GPU use)

ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent 'FFmpeg Frigate/0.12.1-367d724' -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.24:8554/mainstream -r 5 -vf fps=5,scale_vaapi=w=720:h=480,hwdownload,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p prova.mp4

And I get almost zero CPU use.

Thus, one reason why I don't see a GPU load is that I'm using the substream for the detection, which uses very low GPU resources.

But now I am wondering to know what causes the high CPU load. With the command pgrep -a ffmpeg I see that there are several ffmpeg tasks (for each camera). One task is for write into file the mainstream, one task is where I took the code above, one task is the following:

ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:

and the final task is the following

ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 720x480 -i pipe: -f mpegts -s 1080x720 -codec:v mpeg1video -q 8 -bf 0 pipe:

What these last two tasks are needed for? It seems that they don't use hardware acceleration.

Finally, I guess that all this CPU use comes from the detector, which runs on the CPU and not on the integrated GPU, right?

[albertomercurio]

Ok, as I said, the high CPU usage was for the detection model. Changing to the OpenVINO model requires very low CPU usage.

detectors:
  # cpu:
  #   type: cpu
  #   num_threads: 4
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt