NVENC Fails in Kubernetes Pods on all but the last GPU with Driver 570.x or 580.x

[FLM210]

Summary

We are experiencing a critical regression with NVENC hardware encoding when using NVIDIA driver version 570.x in a multi-GPU Kubernetes environment. On a node with four identical GPUs, any containerized application managed by the GPU Operator can only use the NVENC encoder successfully if it is scheduled on the last enumerated GPU (e.g., GPU 3 of 4). Pods scheduled on any other GPU (0, 1, or 2) fail to initialize the encoder.

This issue is a clear regression, as the entire setup works perfectly with the 550.x driver series. Host-level encoding works on all cards, and we have confirmed there is no driver version mismatch between the host and the container. The problem appears to be specific to how the 570.x driver exposes NVENC capabilities to the containerized environment in a multi-GPU configuration.

Environment Details

• Hardware:
  • CPU: AMD Ryzen Threadripper 7970X (32-Cores)
  • GPU: 4 x NVIDIA GeForce RTX 4080 SUPER
  • Motherboard: ASUSTeK Pro WS TRX50-SAGE WIFI
• Software:
  • Orchestrator: Kubernetes
  • GPU Management: NVIDIA GPU Operator
  • Host Driver (Problematic): 570.x (e.g., 570.124.06)
  • Host Driver (Working): 550.x series
  • Container: Using a container with correctly matched user-space libraries for the host driver.
  • Application: An Unreal Engine-based rendering service, and standard ffmpeg.

Steps to Reproduce

1. Configure a Kubernetes node with multiple identical GPUs (e.g., 4x 4080 SUPER) and install NVIDIA host driver 570.x.
2. Deploy the NVIDIA GPU Operator.
3. Deploy a Kubernetes Deployment that requests a single GPU (spec.containers.resources.limits: nvidia.com/gpu: 1).
4. Ensure pods from the Deployment are scheduled on different physical GPUs (e.g., GPU 0, GPU 1, etc.).
5. Inside a pod scheduled on any GPU except the last one, attempt to initialize an NVENC session using any application (ffmpeg, custom code, etc.).

Expected Behavior

The containerized application should be able to successfully initialize the NVENC hardware encoder and perform video encoding, regardless of which physical GPU (0, 1, 2, or 3) is assigned to the pod.

Actual Behavior

1. Consistent Failure on first N-1 GPUs: NVENC initialization fails on pods assigned to GPU 0, GPU 1, and GPU 2.
2. Consistent Success on the last GPU: A pod that is scheduled on GPU 3 works perfectly and can encode video without issue.
3. Application-Agnostic Failure: The issue is not tied to our application. A standard ffmpeg command inside a failing pod reproduces the error perfectly:

   $ ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -t 10 -c:v h264_nvenc -f null -
   ...
   [h264_nvenc @ 0x55de29791c00] OpenEncodeSessionEx failed: unsupported device (2): (no details)
   [h264_nvenc @ 0x55de29791c00] No capable devices found
   Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0

4. Our Unreal Engine application logs corresponding errors:

   LogAVCodecs: Error: Error Creating: Failed to create encoder [NVENC 2]
   LogPixelStreaming: Error: Could not create encoder.
   

Troubleshooting and Analysis

• This is a clear regression, as downgrading the host driver to the 550.x series resolves the issue completely on the exact same hardware and software stack.
• The issue is specific to the container environment. Running ffmpeg with NVENC directly on the host OS works correctly for all 4 GPUs simultaneously.
• The problem is tied to the logical GPU index, not a specific faulty card. Physically swapping the GPUs does not change the behavior; the failure always occurs on the first N-1 logical GPUs.
• Based on this evidence, the behavior strongly suggests a bug in the 570.x driver or a related component of the GPU Operator toolkit. The issue likely lies in the enumeration or initialization process for NVENC capabilities when exposing them to a container in a multi-GPU system.

Workaround

The only known workaround is to downgrade the NVIDIA host driver to a version in the 550.x series.

[elezar]

@FLM210 do you have any indication that this is specific to the containerization of the application? From a containerization perspective, the system-level interfaces between the different driver versions should have remained the same -- unless there are new driver libraries that should be injected by are not. The fact that the last container works seems to speak against this though.

Also, just for more information, are the drivers installed on the host or managed using the GPU Driver Container?

[FLM210]

@elezar
Test Environment

• Our production workload runs in a Kubernetes (K3s) environment, managed by the NVIDIA GPU Operator.
• All of our tests, for both our Unreal Engine application and for ffmpeg, were conducted exclusively from within a Kubernetes pod. We have no data on the driver's behavior on the bare-metal host.
• We have tested two driver installation methods: allowing the GPU Operator to manage the driver deployment, and pre-installing the driver directly on the host OS before installing the Operator. The bug occurs identically with both methods.

Reproducible Findings:

The only variable that changes the outcome is the NVIDIA driver version itself:

• With the working driver 565.57.01 or 550.54.15 installed:

  • An ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -t 10 -c:v h264_nvenc -f null - command using NVENC succeeds in pods scheduled to any of the four GPUs (0, 1, 2, or 3).

• With the problematic driver 570.124.06 or 580.65.06 installed:

  • The ffmpeg -f lavfi -i testsrc=size=1920x1080:rate=30 -t 10 -c:v h264_nvenc -f null - command fails with an NVENC initialization error in pods scheduled to GPUs 0, 1, and 2.
  • The command only succeeds when the pod is scheduled on the last GPU (ID 3).

[vtone-tkay]

I can confirm this bug. It only happens on containers, Docker and Incus, and it's more subtle than "works on the last GPU". It affects encoding AND decoding and the issue is that the container must "have" the last GPU on the host passed through but can use any GPU once that has been done.

We've been able to reproduce the bug using our own code, the ffmpeg hardware decoding functionality and the Nvidia example / Sample code from GitHub.

Worked example using Incus
I have a host with 2 GPU:

• GPU ID 0 a 3060 Ti
• GPU ID 1 a P4 (don't ask).

If I pass only GPU0 through to the container:

container:$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 (UUID: GPU-b1b2fd58-9336-62a1-d84f-b2cd6c7cb29a)

container:$ ls -l /dev/nvidia*

crw-rw-rw- 1 root   root    195,   0 Aug  7 11:31 /dev/nvidia0
crw-rw-rw- 1 nobody nogroup 195, 255 Jul 22 11:15 /dev/nvidiactl
crw-rw-rw- 1 nobody nogroup 511,   0 Jul 22 11:15 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 511,   1 Jul 22 11:15 /dev/nvidia-uvm-tools

Decoding on GPU0 fails with:

ffmpeg:

container:$ ffmpeg -y -vsync "0" -gpu 0 -c:v h264_cuvid -i /tmp/test.mp4 /tmp/foo.yuv

[h264_cuvid @ 0x56d5c06bf800] ctx->cvdl->cuvidGetDecoderCaps(&ctx->caps8) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
[h264_cuvid @ 0x56d5c06bf800] ctx->cvdl->cuvidGetDecoderCaps(&ctx->caps10) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
[h264_cuvid @ 0x56d5c06bf800] ctx->cvdl->cuvidGetDecoderCaps(&ctx->caps12) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_cuvid) -> rawvideo (native))
Error while opening decoder for input stream #0:0 : Generic error in an external library

Nvidia AppDec sample code:

container:$ ./AppDec -i /tmp/test.mp4 -o /tmp/foo.mp4 -gpu 0

HandleVideoSequence : cuvidGetDecoderCaps(&decodecaps) returned error 100 at ../../NvCodec/NvDecoder/NvDecoder.cpp:155

If I now add vslgpu ID1:

container:$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 (UUID: GPU-b1b2fd58-9336-62a1-d84f-b2cd6c7cb29a)
GPU 1: Tesla P4 (UUID: GPU-74fd4e1e-2153-86d9-bf4a-abbc6c2a6be2)

container:$ ls -l /dev/nvidia*
crw-rw-rw- 1 root   root    195,   0 Aug  7 11:31 /dev/nvidia0
crw-rw-rw- 1 root   root    195,   1 Aug 26 12:35 /dev/nvidia1
crw-rw-rw- 1 nobody nogroup 195, 255 Jul 22 11:15 /dev/nvidiactl
crw-rw-rw- 1 nobody nogroup 511,   0 Jul 22 11:15 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 511,   1 Jul 22 11:15 /dev/nvidia-uvm-tools

Everything now works OK.

Nvidia's AppDec sample code shows we're using GPU ID 0:

container:$ ./AppDec -i /tmp/test.mp4-o /tmp/foo.mp4 -gpu 0

GPU in use: NVIDIA GeForce RTX 3060
Decode with demuxing.
[INFO ][12:38:10] Media format: MPEG-PS (MPEG-2 Program Stream) (mpeg)
Session Initialization Time: 28 ms
[INFO ][12:38:10] Video Input Information
        Codec        : MPEG-2
        Frame rate   : 25000/1000 = 25 fps
        Sequence     : Progressive
        Coded size   : [1280, 720]
        Display area : [0, 0, 1280, 720]
        Chroma       : YUV 420
        Bit depth    : 8
Video Decoding Params:
        Num Surfaces : 8
        Crop         : [0, 0, 0, 0]
        Resize       : 1280x720
        Deinterlace  : Weave

Total frame decoded: 2005
Saved in file /tmp/foo.mp4 in nv12 format

The above is an example;

• It can be reproduced on all our multi-GPU hosts
• It happens with any driver in the 570, 575 and 580 series, on both Ubuntu 22.04 and 24.04.

[vtone-tkay]

For reference I believe these issues are also described in "Multi GPU, NVIDIA_VISIBLE_DEVICES and FFMPEG 7 NVDEC errors #1209" -

[vlaminckaxel]

We also have the same problem.
We having it on multi gpu nodes with 2x4090 and 2x 5090.
But it works in multi gpu nodes with v100 cards!

[zexi]

Same here. It only works if I pass through all GPUs.

[ehfd]

https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-565/347361
Brought this up with the driver team.

More things to add are that GStreamer nvcodec plugin is also affected by this issue, and that the only GPU working in my case is /dev/nvidia2, the third GPU out of five (0-4) GPUs.

More issues:
#1209
NVIDIA/k8s-device-plugin#1282

[Zblocker64]

This is what I did to fix this. Hope it helps

NVIDIA/k8s-device-plugin#1282 (comment)