Multiple Nvidia Cards for Hardware Acceleration? #7792

domenicdistefano · 2023-09-14T02:16:33Z

domenicdistefano
Sep 14, 2023

I found a good deal on an Nvidia T400 card I ordered to go alongside my existing P600. I currently have 12 cameras balancing between the iGPU and P600 but the discreet card is nearly maxed out with its 2GB of VRAM. The T400 has 4GB so I would like to add instead of replacing the p600 if possible.

So according to the below examples from the docs enabling both cards in the Docker Compose file seems straight forward. My question however is how exactly do you assign a particular camera stream for detection to a particular NVIDIA GPU?

Its easy to assign / differentiate between the iGP & a single NVIDIA card by adding ether hardware=vaapi or hardware=cuda to the end of the go2rtc section of the ffmpeg config for a particular substream of a camera. Problem is I am not seeing any way on the go2rtc Hardware Acceleration wiki page around how to assign a camera's substream to a particular NVIDA card if more than one is present.

Anyone figured this out?

Frigate Docs:

services:
  frigate:
    ...
    image: ghcr.io/blakeblackshear/frigate:stable
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0'] # this is only needed when using multiple GPUs
              count: 1 # number of GPUs
              capabilities: [gpu]

Docker Docs:
Access specific devices
To allow access only to GPU-0 and GPU-3 devices:

services:
  test:
    image: tensorflow/tensorflow:latest-gpu
    command: python -c "import tensorflow as tf;tf.test.gpu_device_name()"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0', '3']
            capabilities: [gpu]

Answered by NickM-27

Sep 20, 2023

Oh copied an older one, try


go2rtc:
  ffmpeg:
    cuda0: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 0 -rtsp_transport tcp -i {input}"
    cuda1: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {input}"

  streams:
    back_porch:
      - rtsp://admin:redacted10.10.10.71:554/media/video0
    back_porch_sub:
      - "ffmpeg:back_porch#input=cuda1#video=h264#width=1920#height=1080#raw=-fpsmax 10 -gpu 1#hardware=cuda"

View full answer

NickM-27 · 2023-09-14T11:30:59Z

NickM-27
Sep 14, 2023
Collaborator Sponsor

it's possible to use two separate cards for decoding and object detection, I am not aware of a way to use separate cards for separate streams in frigate or in go2rtc via presets.

You can of course do this via manual ffmpeg args

0 replies

domenicdistefano · 2023-09-14T11:40:07Z

domenicdistefano
Sep 14, 2023
Author

So I wonder how how stream decoding would be assigned then between two NVIDIA cards in such a configuration. I will cross post this over on the go2rtc forum & then report back here if they chime in with any additional info.

1 reply

NickM-27 Sep 14, 2023
Collaborator Sponsor

need to use manual args https://saturncloud.io/blog/how-to-specify-the-gpu-for-nvenc-in-ffmpeg/#:~:text=Specifying%20the%20GPU%20for%20nvenc%20in%20FFmpeg,-Now%20that%20we&text=To%20do%20this%2C%20we'll,the%20video%20encoder%20to%20use.

domenicdistefano · 2023-09-17T03:58:14Z

domenicdistefano
Sep 17, 2023
Author

My T400 card arrived and seems to be working fine / recognized in Docker (GPU 0) but despite the reference Nick posted above I still can't get any streams over to the P600 (GPU 1).

Notice the hwaccel_device 1 argument I added in hopes of it redirecting this stream to the P600 (GPU 1). Arg - no dice...

No errors are showing up in either the Frigate or go2rtc logs and and the stream seems to be working fine but just still on GPU 0. At the bottom below I have my updated Docker Compose where I adjusted the devices parameter to now read device_ids: ['0', '1'] instead of count: 1

Any ideeas?

go2rtc:
  streams:
    back_porch:
      - rtsp://admin:redacted@10.10.10.71:554/media/video0
    back_porch_sub:
      "ffmpeg:back_porch#hwaccel_device 1#video=h264#width=3840#height=2160#raw=-fpsmax 5##hardware=cuda"

nvidia-smi

SMI

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA T400 4GB                On  | 00000000:01:00.0 Off |                  N/A |
| 38%   49C    P0              N/A /  31W |   1979MiB /  4096MiB |     14%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Quadro P600                    On  | 00000000:02:00.0 Off |                  N/A |
| 34%   33C    P8              N/A /  N/A |      4MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      8848      C   ffmpeg                                      108MiB |
|    0   N/A  N/A      8851      C   ffmpeg                                      451MiB |
|    0   N/A  N/A      8884      C   ffmpeg                                      451MiB |
|    0   N/A  N/A      8910      C   ffmpeg                                      254MiB |
|    0   N/A  N/A      8924      C   ffmpeg                                      151MiB |
|    0   N/A  N/A      8925      C   ffmpeg                                      151MiB |
|    0   N/A  N/A      8930      C   ffmpeg                                      254MiB |
|    0   N/A  N/A      8957      C   ffmpeg                                      151MiB |
+---------------------------------------------------------------------------------------+

Docker Compose file

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    cap_add:
      - CAP_PERFMON
    runtime: nvidia
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
    shm_size: "768mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /root/docker/appdata/frigate/config.yml:/config/config.yml:rw
      - /root/docker/appdata/frigate/storage:/media/frigate
      - /root/docker/appdata/frigate/go2rtc:/config/
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: frigate
      NVIDIA_VISIBLE_DEVICES: all 
      NVIDIA_DRIVER_CAPABILITIES: all

1 reply

NickM-27 Sep 17, 2023
Collaborator Sponsor

Notice the hwaccel_device 1 argument I added in hopes of it redirecting this stream to the P600 (GPU 1). Arg - no dice...

that's not how that works, you can't just add a # and something after. I'd suggest reading the go2rtc docs, in this case you probably want to use #input

domenicdistefano · 2023-09-17T04:53:02Z

domenicdistefano
Sep 17, 2023
Author

that's not how that works, you can't just add a # and something after. I'd suggest reading the go2rtc docs, in this case you probably want to use #input

That's also what I suspected at first but the docs over on the go2rtc page don't make much sense to me in that they have a separate wiki page explaining hardware acceleration and all of the # parameters you can string together but as you pointed out you can't just "inject" other ffmpeg arguments by sticking a # in front of it. Then the main page doesn't spell out how exactly you use the #input overall and especially when you are using the examples found on the hw accell page. I cross posted this post over there but never got a response...

O well, giving up for now. Suppose I will just pull the P600 and stick with the new T400 since at least I has 2GB more VRAM.

0 replies

domenicdistefano · 2023-09-17T13:34:05Z

domenicdistefano
Sep 17, 2023
Author

OK, trying another approach here - this time taking go2rtc out of the mix.

Above I was focusing on using the second Nvidia GPU for reencoding my sub-stream that's handles by go2rtc. Why not assign the second GPU to handle Frigate specific function?

So farther down in my config where have my global ffmpeg preset I started with this;

ffmpeg:
      hwaccel_args: preset-vaapi # optionally here could also use preset-nvidia-h264

Since Frigate doesn't have a preset for pointing to a second Nvidia GPU I tries this;

ffmpeg:
      hwaccel_args: ffmpeg -hide_banner 0 -c:v h264_nvenc -hwaccel_device 1 -g 50 -profile:v high -level:v auto -preset:v p2 -tune:v ll 1

Unfortunately nothing but vomit in my log... I referred to this Frigate doc for presents for inspiration on how to properly format the above line.

2 replies

NickM-27 Sep 17, 2023
Collaborator Sponsor

hwaccel is used for decoding the stream, h264_nvenc is an encoding arg along with all of the other args that you are using, they are not in the right place.

NickM-27 Sep 17, 2023
Collaborator Sponsor

you could simply use -c:v h264_cuvid -hwaccel_device 1

domenicdistefano · 2023-09-17T13:52:21Z

domenicdistefano
Sep 17, 2023
Author

Odd. I went with your suggestion that less = more and simply used -c:v h264_cuvid -hwaccel_device 1 but all of the decoding got piled onto GPU 0 and not GPU 1 as the command is telling it to do.

First nvidia-smi screenshot below run from the host OS shows all of the processes are now indeed going onto GPU 0 (half were previously directed to the iGPU when using hwaccel_args: preset-vaapi)

The second screenshot is run from within the container verifying it can see both GPUs so no idea why the Frigate ffmpeg decoding isn't going to GPU 1.

1 reply

NickM-27 Sep 17, 2023
Collaborator Sponsor

Maybe try -c:v h264_cuvid -hwaccel_device 1 -gpu 1

domenicdistefano · 2023-09-17T14:38:55Z

domenicdistefano
Sep 17, 2023
Author

One step forward, one step backyards. After adding -gpu 1 nvidia-smi is now showing the ffmpeg processes are being split as expected between the two GPUS as shown in the first screenshot below (good) - but then look at what the logs are spitting out (bad). Then in the second screenshot notice the GUI isn't showing the second Nvidia GPU at all.

2023-09-17 10:23:42.875759715  [2023-09-17 10:23:42] frigate.video                  ERROR   : pool_road: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.875923068  [2023-09-17 10:23:42] frigate.video                  ERROR   : pool_road: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.884830722  [2023-09-17 10:23:42] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.885030539  [2023-09-17 10:23:42] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.896692249  [2023-09-17 10:23:42] frigate.video                  ERROR   : back_yard: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.896870600  [2023-09-17 10:23:42] frigate.video                  ERROR   : back_yard: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.900806636  [2023-09-17 10:23:42] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.900946817  [2023-09-17 10:23:42] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.908900571  [2023-09-17 10:23:42] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.909068177  [2023-09-17 10:23:42] frigate.video                  ERROR   : front_yard: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.917575264  [2023-09-17 10:23:42] frigate.video                  ERROR   : front_door: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.917735977  [2023-09-17 10:23:42] frigate.video                  ERROR   : front_door: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.921013516  [2023-09-17 10:23:42] frigate.video                  ERROR   : driveway_front: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.921166595  [2023-09-17 10:23:42] frigate.video                  ERROR   : driveway_front: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.937649650  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-41: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.937804563  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-41: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.938938590  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-cheltenham: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.939198664  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-cheltenham: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.949577669  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-pool: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.949724656  [2023-09-17 10:23:42] frigate.video                  ERROR   : lpr-pool: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.949868976  [2023-09-17 10:23:42] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.950016039  [2023-09-17 10:23:42] frigate.video                  ERROR   : driveway_pool: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:42.950679799  [2023-09-17 10:23:42] frigate.video                  ERROR   : doorbell: Unable to read frames from ffmpeg process.
2023-09-17 10:23:42.950834336  [2023-09-17 10:23:42] frigate.video                  ERROR   : doorbell: ffmpeg process is not running. exiting capture thread...
2023-09-17 10:23:52.982510089  [2023-09-17 10:23:52] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-17 10:23:52.982578728  [2023-09-17 10:23:52] frigate.record                 ERROR   : list index out of range
2023-09-17 10:23:57.766495267  [2023-09-17 10:23:57] watchdog.pool_road             ERROR   : Ffmpeg process crashed unexpectedly for pool_road.
2023-09-17 10:23:57.766865025  [2023-09-17 10:23:57] watchdog.pool_road             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.767243271  [2023-09-17 10:23:57] ffmpeg.pool_road.detect        ERROR   : rtsp://127.0.0.1:8554/pool_road_sub: Invalid data found when processing input
2023-09-17 10:23:57.778419325  [2023-09-17 10:23:57] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-17 10:23:57.778919544  [2023-09-17 10:23:57] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.779453535  [2023-09-17 10:23:57] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub: Invalid data found when processing input
2023-09-17 10:23:57.796665937  [2023-09-17 10:23:57] watchdog.back_yard             ERROR   : Ffmpeg process crashed unexpectedly for back_yard.
2023-09-17 10:23:57.797225149  [2023-09-17 10:23:57] watchdog.back_yard             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.797447625  [2023-09-17 10:23:57] ffmpeg.back_yard.detect        ERROR   : rtsp://127.0.0.1:8554/back_yard_sub: Invalid data found when processing input
2023-09-17 10:23:57.806074514  [2023-09-17 10:23:57] watchdog.front_yard            ERROR   : Ffmpeg process crashed unexpectedly for front_yard.
2023-09-17 10:23:57.806319411  [2023-09-17 10:23:57] watchdog.front_yard            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.806499958  [2023-09-17 10:23:57] ffmpeg.front_yard.detect       ERROR   : rtsp://127.0.0.1:8554/front_yard_sub: Invalid data found when processing input
2023-09-17 10:23:57.809517330  [2023-09-17 10:23:57] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-09-17 10:23:57.810200931  [2023-09-17 10:23:57] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.810206701  [2023-09-17 10:23:57] ffmpeg.garden.detect           ERROR   : rtsp://127.0.0.1:8554/garden_sub: Invalid data found when processing input
2023-09-17 10:23:57.815271363  [2023-09-17 10:23:57] watchdog.driveway_front        ERROR   : Ffmpeg process crashed unexpectedly for driveway_front.
2023-09-17 10:23:57.815816521  [2023-09-17 10:23:57] watchdog.driveway_front        ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.815977526  [2023-09-17 10:23:57] ffmpeg.driveway_front.detect   ERROR   : rtsp://127.0.0.1:8554/driveway_front_sub: Invalid data found when processing input
2023-09-17 10:23:57.825267517  [2023-09-17 10:23:57] watchdog.front_door            ERROR   : Ffmpeg process crashed unexpectedly for front_door.
2023-09-17 10:23:57.825376407  [2023-09-17 10:23:57] watchdog.lpr-41                ERROR   : Ffmpeg process crashed unexpectedly for lpr-41.
2023-09-17 10:23:57.829921944  [2023-09-17 10:23:57] watchdog.front_door            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.830184674  [2023-09-17 10:23:57] ffmpeg.front_door.detect       ERROR   : rtsp://127.0.0.1:8554/front_door_sub: Invalid data found when processing input
2023-09-17 10:23:57.830271914  [2023-09-17 10:23:57] watchdog.lpr-41                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.830539759  [2023-09-17 10:23:57] ffmpeg.lpr-41.detect           ERROR   : rtsp://127.0.0.1:8554/lpr-41_sub: Invalid data found when processing input
2023-09-17 10:23:57.833677630  [2023-09-17 10:23:57] watchdog.driveway_pool         ERROR   : Ffmpeg process crashed unexpectedly for driveway_pool.
2023-09-17 10:23:57.833682667  [2023-09-17 10:23:57] watchdog.driveway_pool         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-17 10:23:57.833685404  [2023-09-17 10:23:57] ffmpeg.driveway_pool.detect    ERROR   : rtsp://127.0.0.1:8554/driveway_pool_sub: Invalid data found when processing input
2023-09-17 10:23:57.838764500  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2023-09-17 10:23:57.838769580  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
2023-09-17 10:23:57.838772960  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] no frame!
2023-09-17 10:23:57.838776025  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] missing picture in access unit with size 7914
2023-09-17 10:23:57.838778819  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2023-09-17 10:23:57.838862906  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
2023-09-17 10:23:57.838903568  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] no frame!
2023-09-17 10:23:57.838977009  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] missing picture in access unit with size 12718
2023-09-17 10:23:57.839036540  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2023-09-17 10:23:57.839091304  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
2023-09-17 10:23:57.839144609  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] no frame!
2023-09-17 10:23:57.839198985  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] missing picture in access unit with size 7528
2023-09-17 10:23:57.839254065  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2023-09-17 10:23:57.839307528  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
2023-09-17 10:23:57.839361611  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] no frame!
2023-09-17 10:23:57.839422555  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] missing picture in access unit with size 11488
2023-09-17 10:23:57.848535943  [2023-09-17 10:23:57] watchdog.lpr-cheltenham        ERROR   : Ffmpeg process crashed unexpectedly for lpr-cheltenham.
2023-09-17 10:23:57.848541531  [2023-09-17 10:23:57] ffmpeg.lpr-41.record           ERROR   : [h264 @ 0x564f46072c40] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

1 reply

NickM-27 Sep 17, 2023
Collaborator Sponsor

yeah, I think the main thing that is supported is running decoding on GPU 0 and running detection on any GPU.

vajonam · 2023-09-18T15:24:19Z

vajonam
Sep 18, 2023
Sponsor

What you could try is change the order of the devices, for example if you bigger card, is 1, you can swap the order they are presented to the container by

- NVIDIA_VISIBLE_DEVICES=1,0
- CUDA_VISIBLE_DEVICES=1,0

where GPU id 1 is the better faster card, and 0 is the older card that can be used for object detection.

0 replies

domenicdistefano · 2023-09-18T22:59:53Z

domenicdistefano
Sep 18, 2023
Author

Hmm, interesting. Didn't know that parameter existed. Hold that thought for a second as I did more testing and found something interesting even with just a single Nvidia card installed. I want to solve / understand this problem first before introducing the second Nvidia card.

So in my case I am trying to accomplish multiple things in terms of assigning separate tasks to a particular GPU. Fist (irrespective of Frigate) I am configuring got2rtc to reencode my 12 main camera streams to something workable & uniform (1920x1080 10 fps) and present those as my substream to Frigate for detection purposes. With 12 cameras this would totally overpower the iGPU on top of everything else but the Nvidia cards do this with ease.

So far so good. Step 2 is where the trouble starts. One would think that the discreet Nvidia card (single card in this example to make it simple) would also easily handle the Frigate decoding / acceleration also essentially leaving the iGPU idle - doesn't seem to be the case however. In my config if havehwaccel_args: preset-vaapito enable Frigate hardware acceleration which works as expected. When running the log file is stable / no errors and I can see the iGPU is about 32% loaded and the Nvidia card is humming along at 20% load doing its thing. The moment however I switch this to hwaccel_args: preset-nvidia-h264 everything goes to crap and I get a log full of ffmpeg vomit. One would think Frigate would work fine swinging all of its GPU workloads over to the discreet GPU which has plenty of headroom as you would expect but as my testing has shown just by trying to move Frigate decoding over to the Nvidia card the log blows up with errors. I tested this on two completely deferent boxes and combinations of hardware. This drove me crazy for the longest time until I figured out by toggling back to vaapi everything would just work.

Even if I disable the iGPU in the BIOS where the OS only sees the Nvidia card I get the same result which seems to downplay the theory of Frigate getting confused with two GPUIs in play.

So for now my rig is at least working the way I described above by assigning Frigate specific functions to the iGPU and essentially only utilizing the Nvidia for the go2rtc transcoding (which also benefits from decoding I assume when the stream is being read read in).

Although everything is working for now in my case I still want to be able to mix & match for max flexibility. Bottom line is I don't know what's going wrong here as I essentially can't replicate even basic success for using the Nvidia card in the way the official docs specify via the hwaccel_args: preset-nvidia-h264 parameter. My hardware components are all off the shelf and I have rebuilt two different boxes from scratch,

1 reply

NickM-27 Sep 18, 2023
Collaborator Sponsor

I have no issues mixing and matching iGPU and nvidia GPU in frigate. If I had to guess, the GPUs decoder is fully or close to fully utilized from the 10 streams that are being transcoded.

No way to know for sure without logs though.

domenicdistefano · 2023-09-19T00:00:54Z

domenicdistefano
Sep 19, 2023
Author

That's what I first suspected but as you can see it's only 35% of whatever "GPU %" refers to,

Hardware tested = Single Nvidia T-400 (also tried P600 & 1080 Titan) , standard i7-6700 in an HP EliteDesk 800 G2, 6700K in an Asus ROG motherboard, 16GB RAM / 768 dedicated to the container, vanilla Ubuntu Server 22.04 rebuilt from scratch several times.

Not sure what I am doing wrong / special.

1 reply

NickM-27 Sep 19, 2023
Collaborator Sponsor

That's just general GPU usage, that does not show the decoder utilization.

domenicdistefano · 2023-09-19T00:34:17Z

domenicdistefano
Sep 19, 2023
Author

Found this command - even with 12 cameras going (2x @4k, 6x @2560,1440, 3x @1920x1080, & 1x @ 2560x1920) max decode = 62% & max encode = 25%

1 reply

NickM-27 Sep 19, 2023
Collaborator Sponsor

So what errors do you get in frigate

vajonam · 2023-09-19T00:40:46Z

vajonam
Sep 19, 2023
Sponsor

There is hard limit on the number of streams a NVIDIA GPU device can encode/decode

https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

3 replies

vajonam Sep 19, 2023
Sponsor

https://www.elpamsoft.com/?p=Plex-Hardware-Transcoding this also shows you how many streams you can successfully transcode, its not just a %GPU usage consideration.

NickM-27 Sep 19, 2023
Collaborator Sponsor

I run the driver patch that removes the limit on stream encodings. Not sure if there is something similar for decode limit, I don't have enough streams to decode to worry though.

vajonam Sep 19, 2023
Sponsor

At the momeny neither do I so I dont bother with the patch, but was aware if of it, but thought it was windows only.

also

@domenicdistefano run watch -n 2 'nvidia-smi -q -d UTILIZATION' will show you more interesting stats on utilization

domenicdistefano · 2023-09-19T00:55:48Z

domenicdistefano
Sep 19, 2023
Author

The hard limit is for encodes but has been "patched" successfully for ~4 years now. with this patch.

Here are my logs with hwaccel_args: preset-vaapi & hwaccel_args: preset-nvidia-h264

iGPU

2023-09-18 20:43:09.144163813  [INFO] Starting Frigate...
2023-09-18 20:43:09.892463273  [2023-09-18 20:43:09] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-18 20:43:09.985437365  [2023-09-18 20:43:09] peewee_migrate                 INFO    : Starting migrations
2023-09-18 20:43:09.988251731  [2023-09-18 20:43:09] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-18 20:43:10.001325944  [2023-09-18 20:43:09] detector.coral1                INFO    : Starting detection process: 448
2023-09-18 20:43:10.005232732  [2023-09-18 20:43:10] detector.coral2                INFO    : Starting detection process: 450
2023-09-18 20:43:10.005654841  [2023-09-18 20:43:10] frigate.app                    INFO    : Output process started: 452
2023-09-18 20:43:10.010694523  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for pool_road: 458
2023-09-18 20:43:10.015810393  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for back_porch: 462
2023-09-18 20:43:10.020597763  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for back_yard: 465
2023-09-18 20:43:10.020602838  [2023-09-18 20:43:10] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-09-18 20:43:10.024890531  [2023-09-18 20:43:10] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:1
2023-09-18 20:43:10.029078392  [2023-09-18 20:43:10] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-18 20:43:10.029201335  [2023-09-18 20:43:10] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-18 20:43:10.030986865  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for garden: 490
2023-09-18 20:43:10.032003505  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for front_yard: 491
2023-09-18 20:43:10.041425058  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for front_door: 493
2023-09-18 20:43:10.041493260  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for driveway_front: 495
2023-09-18 20:43:10.045981736  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for driveway_pool: 496
2023-09-18 20:43:10.051267472  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for lpr-41: 498
2023-09-18 20:43:10.055404720  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for lpr-cheltenham: 499
2023-09-18 20:43:10.060663650  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for lpr-pool: 501
2023-09-18 20:43:10.065074221  [2023-09-18 20:43:10] frigate.app                    INFO    : Camera processor started for doorbell: 502
2023-09-18 20:43:10.070062900  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for pool_road: 504
2023-09-18 20:43:10.074328434  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for back_porch: 507
2023-09-18 20:43:10.078555199  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for back_yard: 511
2023-09-18 20:43:10.083287270  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for garden: 516
2023-09-18 20:43:10.089924913  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for front_yard: 523
2023-09-18 20:43:10.096777305  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for front_door: 532
2023-09-18 20:43:10.101748006  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for driveway_front: 538
2023-09-18 20:43:10.107985181  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for driveway_pool: 545
2023-09-18 20:43:10.115351577  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for lpr-41: 553
2023-09-18 20:43:10.121490971  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for lpr-cheltenham: 562
2023-09-18 20:43:10.128555756  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for lpr-pool: 568
2023-09-18 20:43:10.137725774  [2023-09-18 20:43:10] frigate.app                    INFO    : Capture process started for doorbell: 577

Nvidia

2023-09-18 20:44:22.215761490  [INFO] Starting Frigate...
2023-09-18 20:44:22.963474332  [2023-09-18 20:44:22] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-18 20:44:23.056474416  [2023-09-18 20:44:23] peewee_migrate                 INFO    : Starting migrations
2023-09-18 20:44:23.059260896  [2023-09-18 20:44:23] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-18 20:44:23.072251430  [2023-09-18 20:44:23] detector.coral1                INFO    : Starting detection process: 448
2023-09-18 20:44:23.076675909  [2023-09-18 20:44:23] detector.coral2                INFO    : Starting detection process: 450
2023-09-18 20:44:23.095275894  [2023-09-18 20:44:23] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:1
2023-09-18 20:44:23.100433695  [2023-09-18 20:44:23] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-09-18 20:44:23.100436804  [2023-09-18 20:44:23] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-18 20:44:23.100440304  [2023-09-18 20:44:23] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-18 20:44:23.105309436  [2023-09-18 20:44:23] frigate.app                    INFO    : Output process started: 454
2023-09-18 20:44:23.105555615  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for pool_road: 462
2023-09-18 20:44:23.105647412  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for back_porch: 476
2023-09-18 20:44:23.105717603  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for back_yard: 489
2023-09-18 20:44:23.105783577  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for garden: 491
2023-09-18 20:44:23.106176324  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for front_yard: 492
2023-09-18 20:44:23.113072652  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for front_door: 494
2023-09-18 20:44:23.113649399  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for driveway_front: 495
2023-09-18 20:44:23.117671717  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for driveway_pool: 497
2023-09-18 20:44:23.121258546  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for lpr-41: 498
2023-09-18 20:44:23.125314838  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for lpr-cheltenham: 499
2023-09-18 20:44:23.129215597  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for lpr-pool: 501
2023-09-18 20:44:23.133466457  [2023-09-18 20:44:23] frigate.app                    INFO    : Camera processor started for doorbell: 503
2023-09-18 20:44:23.137247522  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for pool_road: 504
2023-09-18 20:44:23.141262453  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for back_porch: 507
2023-09-18 20:44:23.145362763  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for back_yard: 511
2023-09-18 20:44:23.149905698  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for garden: 515
2023-09-18 20:44:23.156738548  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for front_yard: 525
2023-09-18 20:44:23.165943486  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for front_door: 534
2023-09-18 20:44:23.172474623  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for driveway_front: 539
2023-09-18 20:44:23.179410330  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for driveway_pool: 546
2023-09-18 20:44:23.186476209  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for lpr-41: 559
2023-09-18 20:44:23.194123596  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for lpr-cheltenham: 564
2023-09-18 20:44:23.202087519  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for lpr-pool: 574
2023-09-18 20:44:23.210156327  [2023-09-18 20:44:23] frigate.app                    INFO    : Capture process started for doorbell: 582
2023-09-18 20:44:28.352432954  [2023-09-18 20:44:28] frigate.video                  ERROR   : doorbell: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.353962859  [2023-09-18 20:44:28] frigate.video                  ERROR   : doorbell: ffmpeg process is not running. exiting capture thread...
2023-09-18 20:44:28.695897582  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.696238866  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.696458344  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.696685768  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.696861557  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.696986823  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697143821  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697275840  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697459552  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697574551  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697784108  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.697899784  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698068239  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698189635  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698387374  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698654043  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698784140  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.698882611  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699022658  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699185935  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699353205  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699534319  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699708600  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699823371  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.699976003  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700093593  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700225576  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700383949  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700541858  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700667639  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700793767  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.700904954  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701045260  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701184687  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701400317  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701516907  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701674392  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701829670  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.701980474  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702111248  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702292845  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702420134  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702538007  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702736524  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.702806650  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703005428  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703377226  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703379569  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703506823  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703640857  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703799390  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.703932900  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.704069455  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.704204615  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.704339573  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.704480243  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.704620241  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.713900288  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.714223901  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.714493065  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.714669212  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.714831879  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.714963731  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715109686  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715241193  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715393997  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715520608  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715667258  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715796841  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.715907027  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.716086204  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.719062588  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720417781  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720420802  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720422561  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720434482  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720436819  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720438728  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720451382  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720453260  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720461093  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720462708  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720464391  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720465998  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720467764  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720469337  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720478586  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720480827  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720483095  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720485139  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720565153  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720634706  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.720693954  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721822109  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721825399  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721827042  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721828693  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721830300  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721831872  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721833456  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721835042  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721836621  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721838137  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721839629  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721841163  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721842935  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721845035  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721847014  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721848828  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721850315  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.721852371  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.747536824  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.747541186  [2023-09-18 20:44:28] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-18 20:44:28.755700252  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.755865198  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756020189  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756165038  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756311448  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756461337  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756628411  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756799949  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.756935311  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.757092947  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.757258267  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.757433770  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.757599383  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.757721856  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758033272  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758096020  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758458622  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758593606  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758794355  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.758951058  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.759018807  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.759116628  [2023-09-18 20:44:28] frigate.video                  ERROR   : driveway_pool: Unable to read frames from ffmpeg process.
2023-09-18 20:44:28.759262003  [2023-09-18 20:44:28] frigate.video                  ERROR   : front_yard: ffmpeg process is not running. exiting capture thread...

2 replies

NickM-27 Sep 19, 2023
Collaborator Sponsor

As the nvidia docs linked by @vajonam show, there is a limit on decoding streams too

domenicdistefano Sep 19, 2023
Author

I am not seeing where there is a limit to NVDEC streams just by looking at Nvidia's official charts. The encoding matrix lists "Max # of
concurrent sessions" but the decoding chart doesn't have a similar column.

This article over from this year @ Tom's Hardware says:

The number of concurrent NVDEC decodes isn't restricted, though if you're after real-time decoding, there are limits to how many streams can be handled at once. Nvidia provides a table showing the relative decode speeds of various GPUs by codec, though it doesn't indicate the resolution of those streams. Still, even second generation Maxwell GPUs could do over 400 fps of decode, enough for at least six simultaneous 1080p (?) streams — assuming you don't run out of VRAM.

What's the difference between "decodes" & "real-time decoding"

Then I found this forum post on the Nvidia dev forum from back in 2017 where someone asked what the number of simultaneous decoders was. An Nvidia moderator replied saying "There is no limit on the number of decoders you can run in parallel. It is limited by availability of system resources."

I have another box in the house where I have a P2000 card hosting video for friends & family where I see up to about a dozen streams going all the time without issue.

domenicdistefano · 2023-09-19T01:03:35Z

domenicdistefano
Sep 19, 2023
Author

Ran watch -n 2 'nvidia-smi -q -d UTILIZATION' as suggested. Max decode = 71% / encode = 28%

3 replies

domenicdistefano Sep 19, 2023
Author

I believe I have figured out what is going on & came up with a workaround at least with a single Nvidia GPU - yes max # of decodes is the issue but am I am not exactly sure how to reverse engineer the max # of decodes my T400 can handle.

So just to recap the issue - with a single T400 initially I could successful offload all of the go2rtc reencoding & decoding work to the Nvidia card via the following for each of my 12 cameras specifying hardware=cuda. However the Frigate logs would blow up if I then enabled Frigate Nvidia Hardware Acceleration via haccel_args: preset-nvidia-h264. The only way to make this configuration work was to use the internal iGPU for Frigate Hardware Acceleration via hwaccel_args: preset-vaapi. The suspected issue is an unknown cap for the number of decodes the T400 could handle.

    back_porch:
      - rtsp://admin:redacted@10.10.10.71:554/media/video0
    back_porch_sub:
      - "ffmpeg:back_porch#video=h264#width=1920#height=1080#raw=-fpsmax 10#hardware=cuda"

The way I proved the # of decodes was indeed limited & validate my particular overall Frigate setup can work with Nvidia hardware acceleration was to flip 5 of the 12 go2rtc workloads to hardware=vaapi. I then flipped the Frigate ffmpeg parameter back to hwaccel_args: preset-nvidia-h264 and the log vomit disappeared. I experimented a bit and found if I set any more than 5 cameras within the go2rtc section to use cuda things would blow up again. So at least for my setup using a single Nvidia GPU + the Intel iGPU I found the happy balance.

    front_door:
      - rtsp://admin:redacted@10.10.10.75:554/media/video0
    front_door_sub:
      - "ffmpeg:front_door#video=h264#width=1920#height=1080#raw=-fpsmax 10#hardware=vaapi

But now I am back to where I was in the first place. I still can't figure out how to add a second Nvidia GPU so I can offload everything to the external GPUs. The easiest solution would be to tell go2rtc to offload to a particular Nvidia card (say 0 or 1) for a particular stream. Cracking the nut of the specific command to do this, if its possible today, is the question.

Now with all of this being said & all things being equal is there actually an ADVANTAGE for Frigate to offload its functions to an Nvidia GPU rather than the Intel iGPU via vaapi in the first place? Or is it simply an issue of balancing the workloads you have between both the Nvidia & iGPU so that one isn't overloaded in terms of decodes, encodes, memory etc. for one's particular setup?

Many thanks to everyone above with all the help. I certainly learned a lot working through this issue.

NickM-27 Sep 19, 2023
Collaborator Sponsor

In my testing I've found my Nvidia GPU to have about 500 ms lower latency than the iGPU when used for decoding for detect in frigate. I use the Nvidia GPU for my high priority streams and iGPU for all the others.

vajonam Sep 19, 2023
Sponsor

Also based on this https://www.elpamsoft.com/?p=Plex-Hardware-Transcoding , which is for plex transcoding, Turing based cards seem to support a maximum of 12 streams.

vajonam · 2023-09-19T14:44:04Z

vajonam
Sep 19, 2023
Sponsor

You "should" be able to specify the gpu with new ffmpeg (both for frigate and go2rtc) preset that includes the gpu paramter. I remeber you did have this on before but then were confused with the ffmpeg issues in the logs. now that you have narrowed down max decodes, you target specific cameras to be decoded by specific GPUs.

~~I dont know if there is a clear advantage between iGPU and NVidia, its on the object detection side, it seems that NVIDA based tensorrt seems to newer models first, like yolov7 etc.. but again this is just based on my experience. ~~

I have multiple Nvidia GPU's but one of them is mining GPU that has encoding/decoding disabled. So I haven't had to target a 2nd GPU for ffmpeg/go2rtc.

14 replies

NickM-27 Sep 19, 2023
Collaborator Sponsor

yeah, the double {{ }} only needs to be done in the streams part

domenicdistefano Sep 19, 2023
Author

Different errors now.

Many thanks for the help - need to run out for a while but will be back soon. Learning a lot.

2023-09-19 15:56:23.689957666  [INFO] Preparing go2rtc config...
2023-09-19 15:56:23.946959592  [WARN] Using go2rtc binary from '/config/go2rtc' instead of the embedded one
2023-09-19 15:56:23.946962992  [INFO] Starting go2rtc...
2023-09-19 15:56:24.005587336  15:56:24.005 INF go2rtc version 1.7.1 linux/amd64
2023-09-19 15:56:24.005832008  15:56:24.005 INF [api] listen addr=0.0.0.0:1984
2023-09-19 15:56:24.006272943  15:56:24.006 INF [rtsp] listen addr=0.0.0.0:8554
2023-09-19 15:56:24.006483800  15:56:24.006 INF [webrtc] listen addr=0.0.0.0:8555/tcp
2023-09-19 15:56:24.638686284  15:56:24.638 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/pool_road?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=3840:2160\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.643249590  15:56:24.643 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/back_porch?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.650896162  15:56:24.650 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/back_yard?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.660679469  15:56:24.660 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/garden?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.665604844  15:56:24.665 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/front_yard?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=3840:2160\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.673508880  15:56:24.673 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/front_door?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.686084329  15:56:24.685 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/driveway_pool?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.689403056  15:56:24.689 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/driveway_front?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.699332238  15:56:24.699 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-cheltenham?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.701267898  15:56:24.701 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-41?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.714169742  15:56:24.714 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-pool?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.724691837  15:56:24.724 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/doorbell?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:24.728792780  15:56:24.728 DBG [exec] run url="exec:ffmpeg -hide_banner -fflags nobuffer -flags low_delay -i http://10.10.10.60/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=dingdong -map 0:v:0? -c:v:0 copy -map 0:a:0? -c:a:0 copy -map 0:a:0? -c:a:1 libopus -application:a:1 lowdelay -frame_duration 20 -min_comp 0 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.066833314  [h264 @ 0x55c1af7720c0] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.131186111  [h264 @ 0x55ef82adb100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.160607983  [h264 @ 0x55bbd61d1100] SEI type 5 size 2216 truncated at 1944
2023-09-19 15:56:25.161303654  [h264 @ 0x55bbd61d1100] SEI type 5 size 2216 truncated at 1928
2023-09-19 15:56:25.172917007  [h264 @ 0x55fa952e1100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.204513775  [h264 @ 0x558af0d70100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.219666597  [h264 @ 0x55bbd61d1100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.228008912  [h264 @ 0x56058855d100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.251064445  [h264 @ 0x55b47ee71100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.296974914      Last message repeated 1 times
2023-09-19 15:56:25.296978398  [h264 @ 0x558af0d70100] slice type 32 too large at 127
2023-09-19 15:56:25.296979740  [h264 @ 0x558af0d70100] decode_slice_header error
2023-09-19 15:56:25.558711339  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-cheltenham?video':
2023-09-19 15:56:25.558725009    Metadata:
2023-09-19 15:56:25.558908505      title           : go2rtc/1.7.1
2023-09-19 15:56:25.558999981    Duration: N/A, start: 0.067000, bitrate: N/A
2023-09-19 15:56:25.559171918    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:25.568014368  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-41?video':
2023-09-19 15:56:25.568023669    Metadata:
2023-09-19 15:56:25.568029447      title           : go2rtc/1.7.1
2023-09-19 15:56:25.568078705    Duration: N/A, start: 0.067000, bitrate: N/A
2023-09-19 15:56:25.568125763    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:25.568429267  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-cheltenham?video) is not a decoding option.
2023-09-19 15:56:25.568920543  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-41?video) is not a decoding option.
2023-09-19 15:56:25.571533293  15:56:25.571 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-41?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.571598811  15:56:25.571 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-cheltenham?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.576018235  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-pool?video':
2023-09-19 15:56:25.576032538    Metadata:
2023-09-19 15:56:25.576042596      title           : go2rtc/1.7.1
2023-09-19 15:56:25.576114528    Duration: N/A, start: 0.066000, bitrate: N/A
2023-09-19 15:56:25.576446698    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:25.576728614  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-pool?video) is not a decoding option.
2023-09-19 15:56:25.579947896  15:56:25.579 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/lpr-pool?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.783912215  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/pool_road?video':
2023-09-19 15:56:25.783927344    Metadata:
2023-09-19 15:56:25.783993359      title           : go2rtc/1.7.1
2023-09-19 15:56:25.784001844    Duration: N/A, start: 0.266667, bitrate: N/A
2023-09-19 15:56:25.784163681    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 3840x2160, 30 fps, 30 tbr, 90k tbn
2023-09-19 15:56:25.785048044  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/pool_road?video) is not a decoding option.
2023-09-19 15:56:25.788887656  15:56:25.788 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/pool_road?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=3840:2160\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.894265343  [h264 @ 0x55ec891670c0] cabac decode of qscale diff failed at 84 68
2023-09-19 15:56:25.894269567  [h264 @ 0x55ec891670c0] error while decoding MB 84 68, bytestream 495062
2023-09-19 15:56:25.894681408  [h264 @ 0x55ec891670c0] concealing 16045 DC, 16045 AC, 16045 MV errors in I frame
2023-09-19 15:56:25.945757596  [h264 @ 0x55ec891670c0] Invalid NAL unit 0, skipping.
2023-09-19 15:56:25.960539294  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/front_yard?video':
2023-09-19 15:56:25.960542329    Metadata:
2023-09-19 15:56:25.960543583      title           : go2rtc/1.7.1
2023-09-19 15:56:25.960559067    Duration: N/A, start: 0.233333, bitrate: N/A
2023-09-19 15:56:25.960586626    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 3840x2160, 30 fps, 30 tbr, 90k tbn
2023-09-19 15:56:25.960719718  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/front_yard?video) is not a decoding option.
2023-09-19 15:56:25.961638007  15:56:25.961 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/front_yard?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=3840:2160\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:25.996544764  [h264 @ 0x55a100fc40c0] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.041799963  [h264 @ 0x558e6084f0c0] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.051911456      Last message repeated 1 times
2023-09-19 15:56:26.051926841  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/back_porch?video':
2023-09-19 15:56:26.051935471    Metadata:
2023-09-19 15:56:26.051994405      title           : go2rtc/1.7.1
2023-09-19 15:56:26.052003294    Duration: N/A, start: 0.280000, bitrate: N/A
2023-09-19 15:56:26.052095373    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.052520584  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/back_porch?video) is not a decoding option.
2023-09-19 15:56:26.056022368  15:56:26.055 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/back_porch?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.062189517      Last message repeated 1 times
2023-09-19 15:56:26.062203689  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/back_yard?video':
2023-09-19 15:56:26.062211553    Metadata:
2023-09-19 15:56:26.062288600      title           : go2rtc/1.7.1
2023-09-19 15:56:26.062603168    Duration: N/A, start: 0.320000, bitrate: N/A
2023-09-19 15:56:26.063112024    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.063669634  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/back_yard?video) is not a decoding option.
2023-09-19 15:56:26.067979677  15:56:26.067 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/back_yard?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.087676743  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/garden?video':
2023-09-19 15:56:26.087689430    Metadata:
2023-09-19 15:56:26.087698369      title           : go2rtc/1.7.1
2023-09-19 15:56:26.087763952    Duration: N/A, start: 0.320000, bitrate: N/A
2023-09-19 15:56:26.087882184    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.088318502  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/garden?video) is not a decoding option.
2023-09-19 15:56:26.092315082  15:56:26.091 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/garden?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.105379741      Last message repeated 1 times
2023-09-19 15:56:26.105394950  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/driveway_pool?video':
2023-09-19 15:56:26.105402913    Metadata:
2023-09-19 15:56:26.105468361      title           : go2rtc/1.7.1
2023-09-19 15:56:26.105822672    Duration: N/A, start: 0.320000, bitrate: N/A
2023-09-19 15:56:26.106920836    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440    Last message repeated 1 times
2023-09-19 15:56:26.106931441  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/driveway_front?video':
2023-09-19 15:56:26.106937721    Metadata:
2023-09-19 15:56:26.106960247      title           : go2rtc/1.7.1
2023-09-19 15:56:26.106968060    Duration: N/A, start: 0.280000, bitrate: N/A
2023-09-19 15:56:26.106976716    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.107853967  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/driveway_front?video) is not a decoding option.
2023-09-19 15:56:26.117489240  15:56:26.113 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/driveway_front?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.117503157  , 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.117516696  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/driveway_pool?video) is not a decoding option.
2023-09-19 15:56:26.121112411  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/front_door?video':
2023-09-19 15:56:26.121122884    Metadata:
2023-09-19 15:56:26.121129019      title           : go2rtc/1.7.1
2023-09-19 15:56:26.121179829    Duration: N/A, start: 0.320000, bitrate: 15:56:26.120 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/driveway_pool?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.121184097  N/A
2023-09-19 15:56:26.121284459    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:26.121596903  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/front_door?video) is not a decoding option.
2023-09-19 15:56:26.124145321  15:56:26.123 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -gpu 1 -rtsp_transport tcp -i rtsp://127.0.0.1:8554/front_door?video -fpsmax 10 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-19 15:56:26.169639802  [h264 @ 0x559b2a942100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.178633974  [h264 @ 0x559b2a942100] slice type 32 too large at -1
2023-09-19 15:56:26.178637009  [h264 @ 0x559b2a942100] decode_slice_header error
2023-09-19 15:56:26.202385427  [h264 @ 0x560d72dd8100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.246446509  [h264 @ 0x564a846ef100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.254541808  [h264 @ 0x564a846ef100] slice type 32 too large at -1
2023-09-19 15:56:26.254546161  [h264 @ 0x564a846ef100] decode_slice_header error
2023-09-19 15:56:26.296422950  [h264 @ 0x5555cb420100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.343523218  [h264 @ 0x556a96aef100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.371976944  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-pool?video':
2023-09-19 15:56:26.371979778    Metadata:
2023-09-19 15:56:26.371981086      title           : go2rtc/1.7.1
2023-09-19 15:56:26.371994523    Duration: N/A, start: 0.066000, bitrate: N/A
2023-09-19 15:56:26.372023383    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:26.372219726  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-pool?video) is not a decoding option.
2023-09-19 15:56:26.372825852  15:56:26.372 WRN [rtsp] error="streams: exec: exit status 1" stream=lpr-pool_sub
2023-09-19 15:56:26.376709377  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-41?video':
2023-09-19 15:56:26.376712356    Metadata:
2023-09-19 15:56:26.376713405      title           : go2rtc/1.7.1
2023-09-19 15:56:26.376714432    Duration: N/A, start: 0.067000, bitrate: N/A
2023-09-19 15:56:26.376727267    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:26.376866163  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-41?video) is not a decoding option.
2023-09-19 15:56:26.377469059  15:56:26.377 WRN [rtsp] error="streams: exec: exit status 1" stream=lpr-41_sub
2023-09-19 15:56:26.391424580  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/lpr-cheltenham?video':
2023-09-19 15:56:26.391427169    Metadata:
2023-09-19 15:56:26.391441360      title           : go2rtc/1.7.1
2023-09-19 15:56:26.391442639    Duration: N/A, start: 0.066000, bitrate: N/A
2023-09-19 15:56:26.391474292    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 30 tbr, 90k tbn
2023-09-19 15:56:26.391598698  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-cheltenham?video) is not a decoding option.
2023-09-19 15:56:26.392145657  15:56:26.392 WRN [rtsp] error="streams: exec: exit status 1" stream=lpr-cheltenham_sub
2023-09-19 15:56:26.417168339  [h264 @ 0x5572a4444100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.461098564  [h264 @ 0x559b2a942100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:26.463395924  [h264 @ 0x564a846ef100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.042828841      Last message repeated 4 times
2023-09-19 15:56:27.042833348  [h264 @ 0x560d72dd8100] sps_id 32 out of range
2023-09-19 15:56:27.060842598      Last message repeated 3 times
2023-09-19 15:56:27.060846432  [h264 @ 0x564a846ef100] slice type 32 too large at -1
2023-09-19 15:56:27.060847784  [h264 @ 0x564a846ef100] decode_slice_header error
2023-09-19 15:56:27.117856776      Last message repeated 2 times
2023-09-19 15:56:27.117861907  [h264 @ 0x559b2a942100] SEI type 5 size 2216 truncated at 1944
2023-09-19 15:56:27.118447191  [h264 @ 0x559b2a942100] SEI type 5 size 2216 truncated at 1928
2023-09-19 15:56:27.126162042      Last message repeated 3 times
2023-09-19 15:56:27.126166247  [h264 @ 0x5572a4444100] slice type 32 too large at -1
2023-09-19 15:56:27.126167902  [h264 @ 0x5572a4444100] decode_slice_header error
2023-09-19 15:56:27.153307273  [h264 @ 0x559b2a942100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.184705423  [h264 @ 0x5572a4444100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.188818462      Last message repeated 3 times
2023-09-19 15:56:27.188822764  [h264 @ 0x560d72dd8100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.248829064  [h264 @ 0x564a846ef100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.573992853      Last message repeated 1 times
2023-09-19 15:56:27.573998749  [h264 @ 0x560d72dd8100] SEI type 5 size 2216 truncated at 1944
2023-09-19 15:56:27.574840264  [h264 @ 0x560d72dd8100] SEI type 5 size 2216 truncated at 1928
2023-09-19 15:56:27.621650642      Last message repeated 5 times
2023-09-19 15:56:27.621654777  [h264 @ 0x556a96aef100] SEI type 5 size 2224 truncated at 1944
2023-09-19 15:56:27.622709617  [h264 @ 0x556a96aef100] SEI type 5 size 2224 truncated at 1928
2023-09-19 15:56:27.626578931  [h264 @ 0x560d72dd8100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.686333321  [h264 @ 0x556a96aef100] Invalid NAL unit 0, skipping.
2023-09-19 15:56:27.765862763      Last message repeated 5 times
2023-09-19 15:56:27.765866501  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/pool_road?video':
2023-09-19 15:56:27.765867890    Metadata:
2023-09-19 15:56:27.765883893      title           : go2rtc/1.7.1
2023-09-19 15:56:27.765886246    Duration: N/A, start: 1.233333, bitrate: N/A
2023-09-19 15:56:27.765914960    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 3840x2160, 30 fps, 30 tbr, 90k tbn
2023-09-19 15:56:27.766049723  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/pool_road?video) is not a decoding option.
2023-09-19 15:56:27.766978719  15:56:27.766 WRN [rtsp] error="streams: exec: exit status 1" stream=pool_road_sub
2023-09-19 15:56:27.769018086      Last message repeated 5 times
2023-09-19 15:56:27.769021646  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/front_yard?video':
2023-09-19 15:56:27.769023922    Metadata:
2023-09-19 15:56:27.769026148      title           : go2rtc/1.7.1
2023-09-19 15:56:27.769040586    Duration: N/A, start: 1.066667, bitrate: N/A
2023-09-19 15:56:27.769070121    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 3840x2160, 30 fps, 30 tbr, 90k tbn
2023-09-19 15:56:27.769255366  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/front_yard?video) is not a decoding option.
2023-09-19 15:56:27.770192153  15:56:27.770 WRN [rtsp] error="streams: exec: exit status 1" stream=front_yard_sub
2023-09-19 15:56:28.053910274      Last message repeated 1 times
2023-09-19 15:56:28.053920909  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/back_porch?video':
2023-09-19 15:56:28.053925695    Metadata:
2023-09-19 15:56:28.053930978      title           : go2rtc/1.7.1
2023-09-19 15:56:28.053982105    Duration: N/A, start: 1.080000, bitrate: N/A
2023-09-19 15:56:28.054156986    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.054644027  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/back_porch?video) is not a decoding option.
2023-09-19 15:56:28.057731355  15:56:28.057 WRN [rtsp] error="streams: exec: exit status 1" stream=back_porch_sub
2023-09-19 15:56:28.101046438  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/driveway_pool?video':
2023-09-19 15:56:28.101055956    Metadata:
2023-09-19 15:56:28.101061487      title           : go2rtc/1.7.1
2023-09-19 15:56:28.101110196    Duration: N/A, start: 1.080000, bitrate: N/A
2023-09-19 15:56:28.101308288    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.101832861  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/driveway_pool?video) is not a decoding option.
2023-09-19 15:56:28.104775621  15:56:28.104 WRN [rtsp] error="streams: exec: exit status 1" stream=driveway_pool_sub
2023-09-19 15:56:28.507145561  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/garden?video':
2023-09-19 15:56:28.507152515    Metadata:
2023-09-19 15:56:28.507193185      title           : go2rtc/1.7.1
2023-09-19 15:56:28.507227404    Duration: N/A, start: 1.520000, bitrate: N/A
2023-09-19 15:56:28.507336411    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.507647626  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/garden?video) is not a decoding option.
2023-09-19 15:56:28.510164418  15:56:28.510 WRN [rtsp] error="streams: exec: exit status 1" stream=garden_sub
2023-09-19 15:56:28.510844374      Last message repeated 6 times
2023-09-19 15:56:28.510850520  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/front_door?video':
2023-09-19 15:56:28.510854240    Metadata:
2023-09-19 15:56:28.510883273      title           : go2rtc/1.7.1
2023-09-19 15:56:28.510907337    Duration: N/A, start: 1.520000, bitrate: N/A
2023-09-19 15:56:28.511006942    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.511279557  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/front_door?video) is not a decoding option.
2023-09-19 15:56:28.513059578  15:56:28.512 WRN [rtsp] error="streams: exec: exit status 1" stream=front_door_sub
2023-09-19 15:56:28.544813869  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/back_yard?video':
2023-09-19 15:56:28.544819110    Metadata:
2023-09-19 15:56:28.544842573      title           : go2rtc/1.7.1
2023-09-19 15:56:28.544845231    Duration: N/A, start: 1.480000, bitrate: N/A
2023-09-19 15:56:28.544905655    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.545080345  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/back_yard?video) is not a decoding option.
2023-09-19 15:56:28.546119785  15:56:28.546 WRN [rtsp] error="streams: exec: exit status 1" stream=back_yard_sub
2023-09-19 15:56:28.582893293      Last message repeated 2 times
2023-09-19 15:56:28.582907340  Input #0, rtsp, from 'rtsp://127.0.0.1:8554/driveway_front?video':
2023-09-19 15:56:28.582913570    Metadata:
2023-09-19 15:56:28.582964906      title           : go2rtc/1.7.1
2023-09-19 15:56:28.582972125    Duration: N/A, start: 1.520000, bitrate: N/A
2023-09-19 15:56:28.583072094    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
2023-09-19 15:56:28.583533437  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/driveway_front?video) is not a decoding option.
2023-09-19 15:56:28.587162164  15:56:28.586 WRN [rtsp] error="streams: exec: exit status 1" stream=driveway_front_sub
2023-09-19 15:56:28.790944052  Input #0, flv, from 'http://10.10.10.60/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=dingdong':
2023-09-19 15:56:28.790953875    Metadata:
2023-09-19 15:56:28.790959274      displayWidth    : 2560
2023-09-19 15:56:28.791009784      displayHeight   : 1920
2023-09-19 15:56:28.791052530    Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
2023-09-19 15:56:28.791201595    Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 19 fps, 20 tbr, 1k tbn
2023-09-19 15:56:28.791383561    Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
2023-09-19 15:56:28.792968048  Stream mapping:
2023-09-19 15:56:28.793031399    Stream #0:0 -> #0:0 (copy)
2023-09-19 15:56:28.793040385    Stream #0:1 -> #0:1 (copy)
2023-09-19 15:56:28.793049593    Stream #0:1 -> #0:2 (aac (native) -> opus (libopus))
2023-09-19 15:56:28.793111488  Press [q] to stop, [?] for help
2023-09-19 15:56:28.795482380  [libopus @ 0x56259b5ca140] No bit rate set. Defaulting to 64000 bps.
2023-09-19 15:56:28.846856529  15:56:28.846 DBG [exec] run launch=4.117981617s
2023-09-19 15:56:28.847436887  Output #0, rtsp, to 'rtsp://127.0.0.1:8554/317a6a632d2fa1c7cec3672343904b36':
2023-09-19 15:56:28.847459001    Metadata:
2023-09-19 15:56:28.847506817      displayWidth    : 2560
2023-09-19 15:56:28.847512502      displayHeight   : 1920
2023-09-19 15:56:28.847558424      encoder         : Lavf59.27.100
2023-09-19 15:56:28.847843470    Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, q=2-31, 19 fps, 20 tbr, 90k tbn
2023-09-19 15:56:28.847981604    Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp (default)
2023-09-19 15:56:28.848186851    Stream #0:2: Audio: opus, 16000 Hz, mono, flt, 64 kb/s
2023-09-19 15:56:28.848196641      Metadata:
2023-09-19 15:56:28.848251537        encoder         : Lavc59.37.100 libopus
2023-09-19 15:56:32.033509704  frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:02.14 bitrate=N/A speed=2.14e+06x    
frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:02.84 bitrate=N/A speed=5.63x    
frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:03.35 bitrate=N/A speed=3.32x    
frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:03.80 bitrate=N/A speed=2.51x    
frame=    1 fps=0.5 q=-1.0 size=N/A time=00:00:04.12 bitrate=N/A speed=2.03x    
frame=   16 fps=6.3 q=-1.0 size=N/A time=00:00:04.89 bitrate=N/A speed=1.93x    
frame=   26 fps=8.5 q=-1.0 size=N/A time=00:00:05.40 bitrate=N/A speed=1.77x    
Input #0, rtsp, from 'rtsp://127.0.0.1:8554/doorbell?video':
2023-09-19 15:56:32.033523861    Metadata:
2023-09-19 15:56:32.033583157      title           : go2rtc/1.7.1
2023-09-19 15:56:32.033620802    Duration: N/A, start: 0.350000, bitrate: N/A
2023-09-19 15:56:32.033923259    Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 20 tbr, 90k tbn
2023-09-19 15:56:32.034461857  Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/doorbell?video) is not a decoding option.
2023-09-19 15:56:32.037634321  15:56:32.037 WRN [rtsp] error="streams: exec: exit status 1" stream=doorbell_sub
2023-09-19 15:56:33.689991862  frame=   37 fps= 10 q=-1.0 size=N/A time=00:00:05.91 bitrate=N/A speed=1.65x    
frame=   49 fps= 12 q=-1.0 size=N/A time=00:00:06.42 bitrate=N/A speed=1.55x    
frame=   58 fps= 12 q=-1.0 size=N/A time=00:00:06.94 bitrate=N/A speed=1.47x    
[INFO] Starting go2rtc healthcheck service...

NickM-27 Sep 19, 2023
Collaborator Sponsor

Codec AVOption gpu (Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.) specified for input file #0 (rtsp://127.0.0.1:8554/lpr-cheltenham?video) is not a decoding option. looks like -gpu is only valid as an encoder option not decoder. In that case try this

go2rtc:
  streams:
    back_porch:
      - rtsp://admin:redacted10.10.10.71:554/media/video0
    back_porch_sub:
      - "ffmpeg:back_porch#video=h264#width=1920#height=1080#raw=-fpsmax 10 -gpu 0#hardware=cuda"

domenicdistefano Sep 19, 2023
Author

No go. Tried specifying GPU1 in my case (#raw=-fpsmax 10 -gpu 1) and it sill pins the stream to GPU 0.

vajonam Sep 20, 2023
Sponsor

Check the mutli-gpu setting in there. https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/ maybe try -hwaccel_device 1 instead of -gpu 1

domenicdistefano · 2023-09-20T01:45:56Z

domenicdistefano
Sep 20, 2023
Author

No workie.... Sort of back to where we started. There seems to be various ways to pass what we want with native ffmpeg but the problem is getting it translated into go2rtc speak. Blows my mind to learn go2rtc allows custom presets but now I am thinking everything in between the #s are just that and you could pass virtually any command by defining a preset. We were close above but just ran out of gas / ideas.

8 replies

vajonam Sep 20, 2023
Sponsor

it needs to be right near the -hwaccel cuda

NickM-27 Sep 20, 2023
Collaborator Sponsor

try this

go2rtc:
  ffmpeg:
    cuda0: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 0 -rtsp_transport tcp -i {{input}}"
    cuda1: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {{input}}"

  streams:
    back_porch:
      - rtsp://admin:redacted10.10.10.71:554/media/video0
    back_porch_sub:
      - "ffmpeg:back_porch#input=cuda1#video=h264#width=1920#height=1080#raw=-fpsmax 10 -gpu 1#hardware=cuda"

domenicdistefano Sep 20, 2023
Author

No such file or directory error again.

2023-09-20 10:19:12.365836723  [INFO] Preparing go2rtc config...
2023-09-20 10:19:12.621330745  [WARN] Using go2rtc binary from '/config/go2rtc' instead of the embedded one
2023-09-20 10:19:12.621334172  [INFO] Starting go2rtc...
2023-09-20 10:19:12.680108695  10:19:12.680 INF go2rtc version 1.7.1 linux/amd64
2023-09-20 10:19:12.680300524  10:19:12.680 INF [api] listen addr=0.0.0.0:1984
2023-09-20 10:19:12.680509003  10:19:12.680 INF [rtsp] listen addr=0.0.0.0:8554
2023-09-20 10:19:12.680627494  10:19:12.680 INF [webrtc] listen addr=0.0.0.0:8555/tcp
2023-09-20 10:19:13.190167417  10:19:13.190 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {rtsp://127.0.0.1:8554/back_porch?video} -fpsmax 10 -gpu 1 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-20 10:19:13.194408300  {rtsp://127.0.0.1:8554/back_porch?video}: No such file or directory
2023-09-20 10:19:13.194897395  10:19:13.194 DBG [exec] run url="exec:ffmpeg -hide_banner -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {rtsp://127.0.0.1:8554/back_porch?video} -fpsmax 10 -gpu 1 -c:v h264_nvenc -g 50 -bf 0 -profile:v high -level:v auto -preset:v p2 -tune:v ll -an -vf \"scale_cuda=1920:1080\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-09-20 10:19:13.198256003  {rtsp://127.0.0.1:8554/back_porch?video}: No such file or directory
2023-09-20 10:19:13.198642643  10:19:13.198 WRN [rtsp] error="streams: exec: exit status 1" stream=back_porch_sub

NickM-27 Sep 20, 2023
Collaborator Sponsor

Oh copied an older one, try


go2rtc:
  ffmpeg:
    cuda0: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 0 -rtsp_transport tcp -i {input}"
    cuda1: "-fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -hwaccel_device 1 -rtsp_transport tcp -i {input}"

  streams:
    back_porch:
      - rtsp://admin:redacted10.10.10.71:554/media/video0
    back_porch_sub:
      - "ffmpeg:back_porch#input=cuda1#video=h264#width=1920#height=1080#raw=-fpsmax 10 -gpu 1#hardware=cuda"

Answer selected by NickM-27

domenicdistefano · 2023-09-20T14:27:28Z

domenicdistefano
Sep 20, 2023
Author

It works!!!!!!!

Hang tight while I put my full config back and see if the steering wheel shakes when I hit 80mph.

4 replies

vajonam Sep 20, 2023
Sponsor

think that gpu 1 there is redundant from old copy paste.

NickM-27 Sep 20, 2023
Collaborator Sponsor

I think you have to set both so the decode and encode are done on the same GPU. Could try without it though.

domenicdistefano Sep 20, 2023
Author

For my single camera test that just worked here is the smi output. Putting the entire config back together and will report back how the whole things looks.

Wed Sep 20 10:26:13 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA T400 4GB                On  | 00000000:01:00.0 Off |                  N/A |
| 38%   34C    P0              N/A /  31W |    144MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Quadro P600                    On  | 00000000:02:00.0 Off |                  N/A |
| 34%   39C    P0              N/A /  N/A |    252MiB /  2048MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    353477      C   ffmpeg                                      140MiB |
|    1   N/A  N/A    353479      C   ffmpeg                                      248MiB |
+---------------------------------------------------------------------------------------+

vajonam Sep 20, 2023
Sponsor

I would add one at time, or maybe 2-3 at a time. to see if you run into any other limits on other cards.

vajonam · 2023-09-20T14:36:13Z

vajonam
Sep 20, 2023
Sponsor

makes me want to order in some P600 GPUs ! nice!

0 replies

vajonam · 2023-09-20T14:41:33Z

vajonam
Sep 20, 2023
Sponsor

Probably should mentioned this earlier, but if you can setup the record streams 'correctly' on the device itself like the IP camera, I use the the #video=copy command to copy over the video stream without any decode-rencode, in go2rtc, but still having a restream available avoiding connecting to the camera many times.

I have setup my camera to the resolution and fps. I want to for the recording.

For the detect stream which only read by frigate I have it doing the GPU decode only.

    pathway:
      - ffmpeg:rtsp://admin:{FRIGATE_RTSP_PASSWORD}@ip_camera02:554/cam/realmonitor?channel=1&subtype=0#video=copy

3 replies

NickM-27 Sep 20, 2023
Collaborator Sponsor

Yeah, no need to do this unless the camera doesn't have a suitable sub stream AND you want to use the substream elsewhere besides frigate.

domenicdistefano Sep 20, 2023
Author

Happy to report I have all 12 cameras running and stable so far. The one downside is that other than the decoder limit VRAM in the end is equally a limiting factor. Take a look at the below spread across the two cards. When utilizing the discreet GPU each decode session takes a boatload of memory. In my case the P600 only has 2GB so I couldn't fit all sessions on it so I had to spill over to the T400. This is OK because it has 4GB of VRAM so the net is I still have 2GB VRAM free to add a few more cameras. @ about $75 for the 4GB T400 on Ebay that's not a bad deal.

One thing I can't understand is why the iGPU only uses a fraction of this amount memory to do the same thing. No way I was eating 5GB of system memory before.

Now the question is weather or not I am any better off using the two discreet GPUs exclusively as opposed to just the T400 and the iGPU. Will see how things work out stability and performance wise over time.

Can't thank you guys enough for seeing this through as I think we figured out something others will be able to utilize going forward.

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA T400 4GB                On  | 00000000:01:00.0 Off |                  N/A |
| 38%   50C    P0              N/A /  31W |   3062MiB /  4096MiB |     15%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Quadro P600                    On  | 00000000:02:00.0 Off |                  N/A |
| 46%   60C    P0              N/A /  N/A |   1822MiB /  2048MiB |     16%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    584245      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584249      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584259      C   ffmpeg                                      386MiB |
|    0   N/A  N/A    584263      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584270      C   ffmpeg                                      394MiB |
|    0   N/A  N/A    584273      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584274      C   ffmpeg                                      451MiB |
|    0   N/A  N/A    584279      C   ffmpeg                                      386MiB |
|    0   N/A  N/A    584307      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584310      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584312      C   ffmpeg                                      140MiB |
|    0   N/A  N/A    584322      C   ffmpeg                                      151MiB |
|    0   N/A  N/A    584324      C   ffmpeg                                      151MiB |
|    0   N/A  N/A    584328      C   ffmpeg                                      151MiB |
|    1   N/A  N/A    584257      C   ffmpeg                                      248MiB |
|    1   N/A  N/A    584266      C   ffmpeg                                      248MiB |
|    1   N/A  N/A    584278      C   ffmpeg                                      248MiB |
|    1   N/A  N/A    584289      C   ffmpeg                                      380MiB |
|    1   N/A  N/A    584291      C   ffmpeg                                      248MiB |
|    1   N/A  N/A    584304      C   ffmpeg                                      445MiB |
+---------------------------------------------------------------------------------------+

NickM-27 Sep 20, 2023
Collaborator Sponsor

One thing I can't understand is why the iGPU only uses a fraction of this amount memory to do the same thing. No way I was eating 5GB of system memory before.

iGPU shares memory with the system, meanwhile when using a discrete GPU the frames have to be held in system memory and in GPU VRAM

domenicdistefano · 2023-09-20T22:29:14Z

domenicdistefano
Sep 20, 2023
Author

One last observation. With one of the Nvidia GPUs + iGPU power consumption for the entire box as about 160w. Now with the addition of a second GPU and the iGPU idle total power draw is down to 111w. Go figure. Adds up for a box running 24x7x365.

0 replies

janstadt · 2024-03-29T18:40:57Z

janstadt
Mar 29, 2024

I dont know if this is related or not, but i have an integrated Intel GPU and just recently added an old AMD Radeon HD 6450 i had laying around in a closet to my machine. According to the docs i would need to change the LIBVA_DRIVER_NAME to radeonsi which i already have set to i965. Is it possible to have a mix of different brand/arch GPUs do video processing?

Right now i have my hw_accel_args set to just preset-intel-qsv-h264 and i did try and expose the newly added card (which is available in the container) as well (/dev/dri/renderD129) but it wouldnt show up in the configs and i dont know nearly enough to figure this out on my own. If anyone has any suggestions/advice that'd be great.

The two cards in the machine are:
CoffeeLake-S GT2 [UHD Graphics 630]
Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM]

2 replies

janstadt Mar 29, 2024

And yes, i know that the Radeon is old and a POS. Figured i'd just try and resurrect some old hardware.

NickM-27 Mar 29, 2024
Collaborator Sponsor

No, it is not possible to have ffmpeg run multiple drivers at the same time to use both for handling video decoding

Multiple Nvidia Cards for Hardware Acceleration? #7792

domenicdistefano Sep 14, 2023

Replies: 21 comments · 48 replies

NickM-27 Sep 14, 2023 Collaborator Sponsor

domenicdistefano Sep 14, 2023 Author

NickM-27 Sep 14, 2023 Collaborator Sponsor

domenicdistefano Sep 17, 2023 Author

NickM-27 Sep 17, 2023 Collaborator Sponsor

domenicdistefano Sep 17, 2023 Author

domenicdistefano Sep 17, 2023 Author

NickM-27 Sep 17, 2023 Collaborator Sponsor

NickM-27 Sep 17, 2023 Collaborator Sponsor

domenicdistefano Sep 17, 2023 Author

NickM-27 Sep 17, 2023 Collaborator Sponsor

domenicdistefano Sep 17, 2023 Author

NickM-27 Sep 17, 2023 Collaborator Sponsor

vajonam Sep 18, 2023 Sponsor

domenicdistefano Sep 18, 2023 Author

NickM-27 Sep 18, 2023 Collaborator Sponsor

domenicdistefano Sep 19, 2023 Author

NickM-27 Sep 19, 2023 Collaborator Sponsor

domenicdistefano Sep 19, 2023 Author

NickM-27 Sep 19, 2023 Collaborator Sponsor

vajonam Sep 19, 2023 Sponsor

vajonam Sep 19, 2023 Sponsor

NickM-27 Sep 19, 2023 Collaborator Sponsor

vajonam Sep 19, 2023 Sponsor

domenicdistefano Sep 19, 2023 Author

NickM-27 Sep 19, 2023 Collaborator Sponsor

domenicdistefano Sep 19, 2023 Author

domenicdistefano Sep 19, 2023 Author

domenicdistefano Sep 19, 2023 Author

NickM-27 Sep 19, 2023 Collaborator Sponsor

vajonam Sep 19, 2023 Sponsor

vajonam Sep 19, 2023 Sponsor

NickM-27 Sep 19, 2023 Collaborator Sponsor

domenicdistefano Sep 19, 2023 Author

NickM-27 Sep 19, 2023 Collaborator Sponsor

domenicdistefano Sep 19, 2023 Author

vajonam Sep 20, 2023 Sponsor

domenicdistefano Sep 20, 2023 Author

vajonam Sep 20, 2023 Sponsor

NickM-27 Sep 20, 2023 Collaborator Sponsor

domenicdistefano Sep 20, 2023 Author

NickM-27 Sep 20, 2023 Collaborator Sponsor

domenicdistefano Sep 20, 2023 Author

vajonam Sep 20, 2023 Sponsor

NickM-27 Sep 20, 2023 Collaborator Sponsor

domenicdistefano Sep 20, 2023 Author

vajonam Sep 20, 2023 Sponsor

vajonam Sep 20, 2023 Sponsor

vajonam Sep 20, 2023 Sponsor

NickM-27 Sep 20, 2023 Collaborator Sponsor

domenicdistefano Sep 20, 2023 Author

NickM-27 Sep 20, 2023 Collaborator Sponsor

domenicdistefano Sep 20, 2023 Author

janstadt Mar 29, 2024

janstadt Mar 29, 2024

NickM-27 Mar 29, 2024 Collaborator Sponsor

domenicdistefano
Sep 14, 2023

Replies: 21 comments 48 replies

NickM-27
Sep 14, 2023
Collaborator Sponsor

domenicdistefano
Sep 14, 2023
Author

NickM-27 Sep 14, 2023
Collaborator Sponsor

domenicdistefano
Sep 17, 2023
Author

NickM-27 Sep 17, 2023
Collaborator Sponsor

domenicdistefano
Sep 17, 2023
Author

domenicdistefano
Sep 17, 2023
Author

NickM-27 Sep 17, 2023
Collaborator Sponsor

NickM-27 Sep 17, 2023
Collaborator Sponsor

domenicdistefano
Sep 17, 2023
Author

NickM-27 Sep 17, 2023
Collaborator Sponsor

domenicdistefano
Sep 17, 2023
Author

NickM-27 Sep 17, 2023
Collaborator Sponsor

vajonam
Sep 18, 2023
Sponsor

domenicdistefano
Sep 18, 2023
Author

NickM-27 Sep 18, 2023
Collaborator Sponsor

domenicdistefano
Sep 19, 2023
Author

NickM-27 Sep 19, 2023
Collaborator Sponsor

domenicdistefano
Sep 19, 2023
Author

NickM-27 Sep 19, 2023
Collaborator Sponsor

vajonam
Sep 19, 2023
Sponsor

vajonam Sep 19, 2023
Sponsor

NickM-27 Sep 19, 2023
Collaborator Sponsor

vajonam Sep 19, 2023
Sponsor

domenicdistefano
Sep 19, 2023
Author

NickM-27 Sep 19, 2023
Collaborator Sponsor

domenicdistefano Sep 19, 2023
Author

domenicdistefano
Sep 19, 2023
Author

domenicdistefano Sep 19, 2023
Author

NickM-27 Sep 19, 2023
Collaborator Sponsor

vajonam Sep 19, 2023
Sponsor

vajonam
Sep 19, 2023
Sponsor

NickM-27 Sep 19, 2023
Collaborator Sponsor

domenicdistefano Sep 19, 2023
Author

NickM-27 Sep 19, 2023
Collaborator Sponsor

domenicdistefano Sep 19, 2023
Author

vajonam Sep 20, 2023
Sponsor

domenicdistefano
Sep 20, 2023
Author

vajonam Sep 20, 2023
Sponsor

NickM-27 Sep 20, 2023
Collaborator Sponsor

domenicdistefano Sep 20, 2023
Author

NickM-27 Sep 20, 2023
Collaborator Sponsor

domenicdistefano
Sep 20, 2023
Author

vajonam Sep 20, 2023
Sponsor

NickM-27 Sep 20, 2023
Collaborator Sponsor

domenicdistefano Sep 20, 2023
Author

vajonam Sep 20, 2023
Sponsor

vajonam
Sep 20, 2023
Sponsor

vajonam
Sep 20, 2023
Sponsor

NickM-27 Sep 20, 2023
Collaborator Sponsor

domenicdistefano Sep 20, 2023
Author

NickM-27 Sep 20, 2023
Collaborator Sponsor

domenicdistefano
Sep 20, 2023
Author

janstadt
Mar 29, 2024

NickM-27 Mar 29, 2024
Collaborator Sponsor