[Beta Support]: NVIDIA Quadro P4000 not supported in latest dev release

[lordnight]

Describe the problem you are having

This change ef07563 seems to have unintentionally or intentionally removed support for older NVIDA GPUs like NVIDIA Quadro P4000. If I run an earlier version before that change, it works as before

Beta Version

0.18-59f570f4360b10389c2bfe3e7ce0bdfd472714bf

Issue Category

Object Detection / Detectors

Frigate config file

N/A

Relevant Frigate log output

frigate  | 2026-03-23 10:59:25.633749097  Process frigate.detector:onnx:
frigate  | 2026-03-23 10:59:25.633750995  Traceback (most recent call last):
frigate  | 2026-03-23 10:59:25.633751693    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
frigate  | 2026-03-23 10:59:25.633752232      self.run()
frigate  | 2026-03-23 10:59:25.633752854    File "/opt/frigate/frigate/object_detection/base.py", line 170, in run
frigate  | 2026-03-23 10:59:25.633753405      detections = object_detector.detect_raw(input_frame)
frigate  | 2026-03-23 10:59:25.633753998                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
frigate  | 2026-03-23 10:59:25.633754559    File "/opt/frigate/frigate/object_detection/base.py", line 101, in detect_raw
frigate  | 2026-03-23 10:59:25.633755096      return self.detect_api.detect_raw(tensor_input=tensor_input)
frigate  | 2026-03-23 10:59:25.633755623             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
frigate  | 2026-03-23 10:59:25.633756121    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 77, in detect_raw
frigate  | 2026-03-23 10:59:25.633756767      tensor_output = self.runner.run({model_input_name: tensor_input})
frigate  | 2026-03-23 10:59:25.633757422                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
frigate  | 2026-03-23 10:59:25.633757961    File "/opt/frigate/frigate/detectors/detection_runners.py", line 246, in run
frigate  | 2026-03-23 10:59:25.633758466      self._session.run_with_iobinding(self._io_binding, ro)
frigate  | 2026-03-23 10:59:25.633759084    File "/usr/local/lib/python3.11/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 407, in run_with_iobinding
frigate  | 2026-03-23 10:59:25.633759569      self._sess.run_with_iobinding(iobinding._iobinding, run_options)
frigate  | 2026-03-23 10:59:25.633760722  RuntimeError: Error in execution: Non-zero status code returned while running Conv node. Name:'/backbone/backbone.0/encoder/encoder/embeddings/patch_embeddings/projection/Conv' Status Message: CUDNN_FE failure 11: CUDNN_BACKEND_API_FAILED ; GPU=0 ; hostname=ubuntu-frigate ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=496 ; expr=s_.cudnn_fe_graph->execute(cudnn_handle, s_.variant_pack, ws.get());

Relevant go2rtc log output (if applicable)

No response

Install method

Docker Compose

docker-compose file or Docker CLI command

N/A

Operating system

Ubuntu

CPU / GPU / Hardware

NVIDIA Quadro P4000

Screenshots

No response

Steps to reproduce

No response

Any other information that may be helpful

No response

[NickM-27]

Yes this is generally expected as we need to support the 50 series GPUs, those that need to support older hardware can build a docker image with an older version.

[lawul]

I get a very similar Error since commit ef07563, with a newer GPU though (NVIDIA RTX PRO 4000 Blackwell SFF Edition).
I was about to create a seperate post, but then saw this one.

My log output:

2026-03-22 23:09:29.921477908 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Conv node. Name:'/model.22/cv3.2/cv3.2.0/conv/Conv' Status Message: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:358 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*) Failed to allocate memory for requested buffer of size 12681216
2026-03-22 23:09:29.921532413  
2026-03-22 23:09:29.945024985  Process frigate.detector:onnx_0:
2026-03-22 23:09:29.945030400  Traceback (most recent call last):
2026-03-22 23:09:29.945031031    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2026-03-22 23:09:29.945031589      self.run()
2026-03-22 23:09:29.945032309    File "/opt/frigate/frigate/object_detection/base.py", line 170, in run
2026-03-22 23:09:29.945032814      detections = object_detector.detect_raw(input_frame)
2026-03-22 23:09:29.945033293                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-22 23:09:29.945037093    File "/opt/frigate/frigate/object_detection/base.py", line 101, in detect_raw
2026-03-22 23:09:29.945039245      return self.detect_api.detect_raw(tensor_input=tensor_input)
2026-03-22 23:09:29.945040616             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-22 23:09:29.945041201    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 77, in detect_raw
2026-03-22 23:09:29.945067443      tensor_output = self.runner.run({model_input_name: tensor_input})
2026-03-22 23:09:29.945068123                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-22 23:09:29.945069198    File "/opt/frigate/frigate/detectors/detection_runners.py", line 246, in run
2026-03-22 23:09:29.945069848      self._session.run_with_iobinding(self._io_binding, ro)
2026-03-22 23:09:29.945078640    File "/usr/local/lib/python3.11/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 407, in run_with_iobinding
2026-03-22 23:09:29.945079172      self._sess.run_with_iobinding(iobinding._iobinding, run_options)
2026-03-22 23:09:29.945083159  RuntimeError: Error in execution: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:129 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:121 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, common::Status> = void] CUDA failure 901: operation failed due to a previous error during capture ; GPU=0 ; hostname=6314079038e1 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_graph.cc ; line=61 ; expr=cudaStreamEndCapture(stream_, &graph); 

VRAM was completely free when starting Frigate (regarding "Failed to allocate memory" in first line).

Do you have any quick ideas/suggestions? If this is something different and more info is needed, I will create a seperate post.

[NickM-27]

Oh this seems like the same issue I've seen reported elsewhere and is unrelated to support for the GPU. Looks like we have a bad combo of libraries that can cause failures loading a model.