Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Immich defaults to software encoding for remaining videos after a hardware accelerated encode fails. #9505

Closed
1 of 3 tasks
watn3y opened this issue May 15, 2024 · 2 comments · Fixed by #9452
Closed
1 of 3 tasks

Immich defaults to software encoding for remaining videos after a hardware accelerated encode fails. #9505

watn3y opened this issue May 15, 2024 · 2 comments · Fixed by #9452

Comments

@watn3y
Copy link

watn3y commented May 15, 2024

The bug

I noticed that after a Video failed to encode with the av1_qsv encoder, Immich defaults to libsvtav1 for all remaining encodes in a queue. In this case, both encoders failed to encode a Video because the frame rate was higher than 240fps. I see no reason why Immich wouldn't try av1_qsv again for the next video in the queue.

The OS that Immich Server is running on

Docker 26.1.2 on Debian Bookworm

Version of Immich Server

v1.105.1

Version of Immich Mobile App

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env

    depends_on:
      - redis
      - database
    restart: unless-stopped
    labels:
      traefik.http.services.immich.loadbalancer.server.port: 3001
      traefik.http.routers.immich.rule: Host(`photos.watn3y.de`)
    networks:
      expose:
      internal:
      
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    devices:
      - /dev/dri:/dev/dri
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: unless-stopped
    networks:
      internal:
      
  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    device_cgroup_rules:
      - "c 189:* rmw"
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /dev/bus/usb:/dev/bus/usb

      - model-cache:/cache
    env_file:
      - stack.env
    restart: unless-stopped
    networks:
      internal:
      
  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: unless-stopped
    networks:
      internal:
      
  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: unless-stopped
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    networks:
      internal:
      
volumes:
  model-cache:

networks:
  expose:
    external: true
  internal:
    external: false

Your .env content

not relevant

Reproduction steps

1. Wait until a hardware accelerated video encode fails for any reason.
2. Any further encodes in the queue will now default to software encoding.

Relevant log output

[Nest] 7  - 05/15/2024, 4:13:56 AM     LOG [ImmichMicroservices] [MediaService] Started encoding video cde9f068-c5fc-4397-8e1f-e146fee677e3 {"inputOptions":["-init_hw_device qsv=hw","-filter_hw_device hw"],"outputOptions":["-c:v av1_qsv","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-bf 7","-refs 5","-g 256","-v verbose","-vf format=nv12,hwupload=extra_hw_frames=64","-preset 3","-global_quality 35"],"twoPass":false}
[Nest] 7  - 05/15/2024, 4:13:57 AM     LOG [ImmichMicroservices] [MediaService] Successfully encoded cde9f068-c5fc-4397-8e1f-e146fee677e3
[Nest] 7  - 05/15/2024, 4:13:57 AM     LOG [ImmichMicroservices] [MediaService] Started encoding video 8a059a4e-07c7-4380-a2bf-754b71f6c539 {"inputOptions":["-init_hw_device qsv=hw","-filter_hw_device hw"],"outputOptions":["-c:v av1_qsv","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-bf 7","-refs 5","-g 256","-v verbose","-vf format=nv12,hwupload=extra_hw_frames=64","-preset 3","-global_quality 35"],"twoPass":false}
[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [MediaRepository] ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x55b88180300] Trying to use DRM render node for device 0, with matching kernel driver (i915).
[AVHWDeviceContext @ 0x55b88180300] libva: VA-API version 1.21.0
[AVHWDeviceContext @ 0x55b88180300] libva: User requested driver 'iHD'
[AVHWDeviceContext @ 0x55b88180300] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x55b88180300] libva: Found init function __vaDriverInit_1_21
[AVHWDeviceContext @ 0x55b88180300] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55b88180300] Initialised VAAPI connection: version 1.21
[AVHWDeviceContext @ 0x55b88180300] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 24.1.5 (8068c2e).
[AVHWDeviceContext @ 0x55b88180300] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x55b88180200] Use Intel(R) oneVPL to create MFX session, API version is 2.10, the required implementation version is 1.3
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55b88180200] Initialize MFX session: implementation version is 2.10
[h264 @ 0x55b88211d80] Reinit context to 1440x1088, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'upload/library/watn3y/2022/06/14/20220614_132711_10.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2022-06-14T11:27:11.000000Z
    location        : +51.3176+009.4978/
    location-eng    : +51.3176+009.4978/
    com.android.version: 12
  Duration: 00:00:04.00, start: 0.000000, bitrate: 78638 kb/s
  Stream #0:0[0x1](eng): Video: h264 (Baseline), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, progressive, left), 1440x1080 (1440x1088), 78568 kb/s, 90 fps, 45k tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2022-06-14T11:27:43.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of -90.00 degrees
  Stream #0:1[0x2](eng): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 64 kb/s (default)
    Metadata:
      creation_time   : 2022-06-14T11:27:43.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> av1 (av1_qsv))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x55b88213180] Reinit context to 1440x1088, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x55b881f5b80] w:1440 h:1080 pixfmt:yuv420p tb:1/90000 fr:45000/1 sar:0/1
[auto_scale_0 @ 0x55b881f5f40] w:iw h:ih flags:'' interl:0
[transpose @ 0x55b881f5c40] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'transpose'
[auto_scale_0 @ 0x55b881f5f40] w:1440 h:1080 fmt:yuv420p sar:0/1 -> w:1440 h:1080 fmt:nv12 sar:0/1 flags:0x00000004
[transpose @ 0x55b881f5c40] w:1440 h:1080 dir:1 -> w:1080 h:1440 rotation:clockwise vflip:0
[AVHWDeviceContext @ 0x55b88185040] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 24.1.5 (8068c2e).
[AVHWDeviceContext @ 0x55b88185040] Driver not found in known nonstandard list, using standard behaviour.
[auto_scale_0 @ 0x55b881f5f40] w:1440 h:1080 fmt:yuv420p sar:0/1 -> w:1440 h:1080 fmt:nv12 sar:0/1 flags:0x00000004
    Last message repeated 2 times
[AVHWFramesContext @ 0x55b88127740] Use Intel(R) oneVPL to create MFX session, API version is 2.10, the required implementation version is 2.10
[AVHWFramesContext @ 0x55b88127740] Initialize MFX session: implementation version is 2.10
[av1_qsv @ 0x55b88211180] Using input frames context (format qsv) with av1_qsv encoder.
[av1_qsv @ 0x55b88211180] Encoder: input is video memory surface
[av1_qsv @ 0x55b88211180] Use Intel(R) oneVPL to create MFX session with the specified MFX loader
[av1_qsv @ 0x55b88211180] Using the intelligent constant quality (ICQ) ratecontrol method
[av1_qsv @ 0x55b88211180] Current frame rate is unsupported
[av1_qsv @ 0x55b88211180] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
[vost#0:0/av1_qsv @ 0x55b881d5280] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x55b88208240] Statistics: 0 bytes written, 0 seeks, 0 writeouts
Terminating demuxer thread 0
[AVIOContext @ 0x55b88206bc0] Statistics: 5101156 bytes read, 2 seeks
Conversion failed!

[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [MediaService] Error: ffmpeg exited with code 1: Conversion failed!

[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [MediaService] Error occurred during transcoding. Retrying with QSV acceleration disabled.
[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [MediaRepository] ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[h264 @ 0x2741c1f0580] Reinit context to 1440x1088, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'upload/library/watn3y/2022/06/14/20220614_132711_10.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2022-06-14T11:27:11.000000Z
    location        : +51.3176+009.4978/
    location-eng    : +51.3176+009.4978/
    com.android.version: 12
  Duration: 00:00:04.00, start: 0.000000, bitrate: 78638 kb/s
  Stream #0:0[0x1](eng): Video: h264 (Baseline), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, progressive, left), 1440x1080 (1440x1088), 78568 kb/s, 90 fps, 45k tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2022-06-14T11:27:43.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of -90.00 degrees
  Stream #0:1[0x2](eng): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 64 kb/s (default)
    Metadata:
      creation_time   : 2022-06-14T11:27:43.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
[vost#0:0 @ 0x2741c133780] Matched encoder 'libsvtav1' for codec 'av1'.
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> av1 (libsvtav1))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x2741c1f3180] Reinit context to 1440x1088, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x2741c1b5280] w:1440 h:1080 pixfmt:yuv420p tb:1/90000 fr:45000/1 sar:0/1
[transpose @ 0x2741c1b5340] w:1440 h:1080 dir:1 -> w:1080 h:1440 rotation:clockwise vflip:0
Svt[info]: -------------------------------------------
Svt[info]: SVT [version]:	SVT-AV1 Encoder Lib v2.0.0
Svt[info]: SVT [build]  :	GCC 12.2.0	 64 bit
Svt[info]: LIB Build date: Mar 24 2024 15:35:37
Svt[info]: -------------------------------------------
Svt[warn]: Film grain denoise apply signal is going to be ignored when film grain is off.
Svt[error]: Instance 1: The maximum allowed frame rate is 240 fps
[libsvtav1 @ 0x2741c1f2180] Error setting encoder parameters: bad parameter (0x80001005)
[vost#0:0/libsvtav1 @ 0x2741c133780] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x2741c5902c0] Statistics: 0 bytes written, 0 seeks, 0 writeouts
Terminating demuxer thread 0
[AVIOContext @ 0x2741c260180] Statistics: 5101156 bytes read, 2 seeks
Conversion failed!

[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [JobService] Unable to run job handler (videoConversion/video-conversion): Error: ffmpeg exited with code 1: Conversion failed!

[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [JobService] Error: ffmpeg exited with code 1: Conversion failed!

    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/processor.js:182:22)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7  - 05/15/2024, 4:13:58 AM   ERROR [ImmichMicroservices] [JobService] Object:
{
  "id": "8a059a4e-07c7-4380-a2bf-754b71f6c539",
  "source": "upload"
}

[Nest] 7  - 05/15/2024, 4:13:58 AM     LOG [ImmichMicroservices] [MediaService] Started encoding video a1e4554e-6143-4a18-ad37-7b7217f218cd {"inputOptions":[],"outputOptions":["-c:v av1","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf format=yuv420p","-preset 6","-crf 35"],"twoPass":false}
[Nest] 7  - 05/15/2024, 4:14:19 AM     LOG [ImmichMicroservices] [MediaService] Successfully encoded a1e4554e-6143-4a18-ad37-7b7217f218cd

Additional information

No response
@watn3y watn3y changed the title Immich defaults to software encoding after a single hardware accelerated encode fails. Immich defaults to software encoding for remaining videos after a hardware accelerated encode fails. May 15, 2024
@mertalev
Copy link
Contributor

This is a regression since we now reuse the config object instead of creating a new one each time. So once we mutate it to have acceleration disabled, it will affect later jobs as well. It should be a simple fix.

@mertalev
Copy link
Contributor

Fixed via #9452

@mertalev mertalev linked a pull request May 16, 2024 that will close this issue
feat(server): fully accelerated nvenc #9452
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(server): fully accelerated nvenc immich-app/immich
2 participants
@watn3y @mertalev