Videos: Improve Nvidia GPU Support

[lastzero]

There may be better ways to enable support for Nvidia GPUs than installing the distribution's default drivers, since Nvidia offers a Docker runtime/integration:

• Videos: Official Nvidia transcoding suppport #2125 (comment)

Acceptance Criteria:

• Validate the approach: Does it really work? Does it reduce the CPU load?
• Document compatibility constraints: What hardware/software will work? What will not work? Windows? Linux? macOS?
• Document all the steps required so that other contributors and our team can get transcoding to work as expected
• Create an example Docker config that we can share at https://dl.photoprism.app/docker/

[mlapaglia]

Validate the approach: Does it really work? Does it reduce the CPU load?

This requires the nvidia drivers to be installed on the docker host. The drivers include all of the libraries needed for decoding and encoding, tensorflow, etc.

I tested performance by removing PHOTOPRISM_INIT from the docker arguments and add the following arguments

docker run -d
-e 'NVIDIA_VISIBLE_DEVICES'='all'
-e 'NVIDIA_DRIVER_CAPABILITIES'='all'
-e 'PHOTOPRISM_FFMPEG_ENCODER'='nvidia'
--runtime=nvidia 
'photoprism/photoprism'

Once running I executed a transcode manually from the console. I took this command from the logs when I clicked on the video through the UI.
/usr/bin/ffmpeg -i /photoprism/originals/2022/08/20220819_223142_5D930210.mp4 -pix_fmt yuv420p -c:v h264_nvenc -c:a aac -preset 15 -pixel_format yuv420p -gpu any -vf format=yuv420p -rc:v constqp -cq 0 -tune 2 -r 30 -b:v 50M -profile:v 1 -coder:v 1 -f mp4 -y /photoprism/storage/sidecar/2022/08/20220819_223142_5D930210.mp4.avc

Notice transcoding speed at 1.05x. It appears that ffmpeg doesn't try to use the GPU at all.

Adding -hwaccel auto to the arguments /usr/bin/ffmpeg -hwaccel auto -i /photoprism/originals/2022/08/20220819_223142_5D930210.mp4 -pix_fmt yuv420p -c:v h264_nvenc -c:a aac -preset 15 -pixel_format yuv420p -gpu any -vf format=yuv420p -rc:v constqp -cq 0 -tune 2 -r 30 -b:v 50M -profile:v 1 -coder:v 1 -f mp4 -y /photoprism/storage/sidecar/2022/08/20220819_223142_5D930210.mp4.avc

Notice transcoding speed 1.98x and significantly lower CPU usage. Also, nvidia-smi shows an active transcode running:

[graciousgrey]

Thanks for testing ❤️

[LordNex]

Question. Will this run natively on a Jetson Nano 4gig?

Honestly I'm trying to be able to use it with Home Assistant, DoubleTake, and Frigate. Using PhotoPrism as the data source for the faces I know for training.

I have requests with those parties to hopefully tie into here based on url searches as well.

My whole approach is to have a video doorbell that will recognize anyone that I have taken a picture of and labeled and trigger an automation. I have most of it down. Now it's just trying to get your data into DoubleTake

[lastzero]

It will run on a Jetson, but you have to compile it and manually install the dependencies incl TensorFlow libs if you don't want to use Docker (which includes all of this in a single image).

[LordNex]

It will run on a Jetson, but you have to compile it and manually install the dependencies incl TensorFlow libs if you don't want to use Docker (which includes all of this in a single image).

No I'd prefer to just load the DeepStack base image minus running DeepStack and then run it under docker compose. That way I know the dependencies should already be there for TensorFlow and the CUDA drivers. Should logical to you?

[lastzero]

You need the right TensorFlow libs inside the Docker image unless you only want hardware video transcoding.

[lastzero]

https://dl.photoprism.app/tensorflow/nvidia-jetson/

[ngoldack]

Hi, I still have issues getting this running.
I followed @mlapaglia steps, but it won't work.
I'm running Unraid with an Nvidia T600 installed.

When running nvidia-smi it shows Cuda installed:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA T600         Off  | 00000000:01:00.0 Off |                  N/A |
| 34%   37C    P0    N/A /  41W |      0MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------

However when I start a transcode with /usr/bin/ffmpeg -i originals/2022/03/20220307_022629_7792822B.mp4 -pix_fmt yuv420p -c:v h264_nvenc -c:a aac -preset 15 -pixel_format yuv420p -gpu any -vf format=yuv420p -rc:v constqp -cq 0 -tune 2 -r 30 -b:v 50M -profile:v 1 -coder:v 1 -f mp4 -y /test.avc
I get the following error:

ffmpeg version 5.1-3 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-1)
  configuration: --prefix=/usr --extra-version=3 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=3 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libaribb24 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc --enable-libsmbclient
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e8abd09640] st: 0 edit list: 1 Missing key frame while searching for timestamp: 2000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e8abd09640] st: 0 edit list 1 Cannot find an index entry before timestamp: 2000.
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'originals/2022/03/20220307_022629_7792822B.mp4':
  Metadata:
    major_brand     : XAVC
    minor_version   : 17440767
    compatible_brands: XAVCmp42iso2nras
    creation_time   : 2022-03-07T02:26:29.000000Z
  Duration: 00:00:27.36, start: 0.000000, bitrate: 107968 kb/s
  Stream #0:0[0x1](und): Video: hevc (Rext) (hvc1 / 0x31637668), yuv422p10le(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 97160 kb/s, 50 fps, 50 tbr, 50k tbn (default)
    Metadata:
      creation_time   : 2022-03-07T02:26:29.000000Z
      handler_name    : Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : HEVC Coding
    Side data:
      displaymatrix: rotation of -90.00 degrees
  Stream #0:1[0x2](und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2022-03-07T02:26:29.000000Z
      handler_name    : Sound Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](und): Data: none (rtmd / 0x646D7472), 4505 kb/s (default)
    Metadata:
      creation_time   : 2022-03-07T02:26:29.000000Z
      handler_name    : Timed Metadata Media Handler
      timecode        : 00:01:05:14
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (pcm_s16be (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_nvenc @ 0x55e8abd46400] Cannot load libcuda.so.1
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

Any suggestions?

Edit:
Usage of --runtime=nvidia in other docker containers, such as Jellyfin, works.

[mlapaglia]

do you also have the

-e 'NVIDIA_VISIBLE_DEVICES'='all'
-e 'NVIDIA_DRIVER_CAPABILITIES'='all'
-e 'PHOTOPRISM_FFMPEG_ENCODER'='nvidia'

flags set?

[ngoldack]

do you also have the

-e 'NVIDIA_VISIBLE_DEVICES'='all'
-e 'NVIDIA_DRIVER_CAPABILITIES'='all'
-e 'PHOTOPRISM_FFMPEG_ENCODER'='nvidia'

flags set?

Yes!

[mlapaglia]

ok try the command i used /usr/bin/ffmpeg -hwaccel auto -i /photoprism/originals/2022/08/20220819_223142_5D930210.mp4 -pix_fmt yuv420p -c:v h264_nvenc -c:a aac -preset 15 -pixel_format yuv420p -gpu any -vf format=yuv420p -rc:v constqp -cq 0 -tune 2 -r 30 -b:v 50M -profile:v 1 -coder:v 1 -f mp4 -y /photoprism/storage/sidecar/2022/08/20220819_223142_5D930210.mp4.avc

maybe try a different file to transcode?