All GPU interrupts arrive on MSI-X vector 0 of 8, so one CPU core limits concurrent NVDEC decode per GPU

[zyoNoob]

NVIDIA Open GPU Kernel Modules Version

595.58.03 (packaged release build, "NVIDIA UNIX Open Kernel Module for x86_64 595.58.03 Release Build", Tue Mar 17 2026)

Proprietary driver

Not tested. This GPU (RTX PRO 6000 Blackwell) is supported only by the open kernel modules. The code paths cited below are in kernel-open/.

Operating System and Version

Ubuntu 24.04.4 LTS, running as a KVM guest with the GPUs passed through (PCIe passthrough).

Kernel Release

6.8.0-106-generic (Ubuntu stock kernel, stable release)

Hardware: GPU

• Test system: 4 × NVIDIA RTX PRO 6000 Blackwell Server Edition, 112 vCPUs
• Production systems: 1 × the same GPU per VM, 28 vCPUs, same driver

Describe the bug

Under heavy NVDEC load (hundreds of concurrent H.264/H.265 decode sessions on one GPU), the GPU raises about 1.2 interrupts per decoded frame (all GPU work combined). All of them arrive on MSI-X vector 0, although the driver allocates 8 vectors per GPU. That one vector is serviced on one CPU, and the hard-IRQ handler takes 53-56 µs per interrupt. At about 7,700 interrupts/s the handler uses 41 % of that CPU. With 1080p25 streams the CPU saturates at about 300 streams per GPU, while NVDEC still has capacity left. Changing /proc/irq/<n>/smp_affinity_list moves the whole load to another CPU but cannot spread it.

1. The driver allocates 8 vectors per GPU, but only vector 0 carries traffic. The device advertises a 9-entry MSI-X table (lspci -vvv: MSI-X: Enable+ Count=9), and nv_init_msix() (kernel-open/nvidia/nv-msi.c) enables NV_RM_MAX_MSIX_LINES = 8 of them. Counts from /proc/interrupts after 40 days of uptime on the test system:

GPU	Vector 0	Vector 1	Vector 2	Vectors 3-7
0 (decode-heavy)	526,324,723	0	92	0
1	26,542,732	0	32	0
2	18,496,107	0	42	0
3	16,510,121	0	31	0

2. The vectors could not run in parallel anyway. nvidia_isr_msix() (kernel-open/nvidia/nv.c) takes the per-device spinlock msix_isr_lock around nvidia_isr(). The comment there says that running the vectors in parallel on different CPUs "is not currently supported by nvidia_isr() and its children". So per-GPU interrupt servicing is limited to one CPU in two ways: by how interrupts are routed, and by this lock.

The same lock and the same 8-vector cap are present in the latest tag, 620.70.06 (we have not re-tested on that driver).

Measurements (one GPU; the application decodes with GStreamer nvh264dec / nvh265dec and also runs TensorRT on the same GPU). Hard-IRQ time comes from bpftrace on irq:irq_handler_entry / irq_handler_exit for vector 0; busy = handler time ÷ wall time on its CPU.

Concurrent streams	Decoded frames/s	Interrupts/s	Interrupts per frame	Handler CPU busy (mean / max)	Mean handler time
275, mixed (H.264/H.265, 15-30 fps, mostly 960×1088)	5,483	6,573	1.20	36.5 % / 39.1 %	56 µs
313, same mix	6,297	7,696	1.22	41.0 % / 43.7 %	53 µs
280 × H.265 1080p25	~7,000	not recorded	—	90.9 %	not recorded
300 × H.265 1080p25	~7,500	not recorded	—	100 % (saturated)	not recorded

In the 313-stream run NVDEC utilisation was 53 % (GPU utilisation 97.5 %). The 1080p25 streams cost more interrupt time per frame than the mixed set; we have not found out why.

Impact:

• The ceiling is one CPU per GPU, whatever the host's core count or the GPU's NVDEC capacity.
• In a VM that CPU is a vCPU the guest scheduler also gives to application threads. We saw a storage controller's IRQ land on the same CPU, which adds to the load.

To Reproduce

Preconditions: one GPU, driver 595.58.03, GStreamer 1.24 or later with the nvcodec plugin, an H.265 1080p25 clip at least 10 minutes long.

1. Start N real-time decode sessions on one GPU (N = 100, 200, 300). Each session runs:

   gst-launch-1.0 filesrc location=clip_1080p25.mp4 ! qtdemux ! h265parse ! nvh265dec ! fakesink sync=true

2. Show the interrupt count per vector for that GPU (replace the bus ID):

   grep 'MSIX-0000:07:00.0' /proc/interrupts | awk '{s=0; for(i=2;i<=NF;i++) if($i ~ /^[0-9]+$/) s+=$i; else break; print $1, $(NF-1), s}'

   Run it twice, 10 s apart. Only vector 0 grows.
3. Find vector 0's CPU (cat /proc/irq/<irq>/effective_affinity_list), then watch it with mpstat -P <cpu> 5 (%irq). Measure the handler time with:

   sudo bpftrace -e '
   tracepoint:irq:irq_handler_entry /args->irq == IRQ/ { @t[cpu] = nsecs; }
   tracepoint:irq:irq_handler_exit  /args->irq == IRQ && @t[cpu]/ {
     @busy_ns = sum(nsecs - @t[cpu]); @n = count(); delete(@t[cpu]); }
   interval:s:10 { print(@n); print(@busy_ns); clear(@n); clear(@busy_ns); }'

   (busy % = @busy_ns ÷ 10^10 × 100)
4. Watch NVDEC with nvidia-smi dmon -s u.
5. Write a different CPU to /proc/irq/<irq>/smp_affinity_list. The whole load moves to that one CPU.

Expected: with 8 MSI-X vectors allocated, interrupt load from different engines or sessions spreads over several vectors and CPUs, as with other multi-queue PCIe devices (NVMe, multi-queue NICs), and decode capacity per GPU is limited by NVDEC, not by one CPU.

Actual: about 100 % of interrupts arrive on vector 0. The busy share of one CPU rises with the number of streams, and at about 300 × 1080p25 streams it reaches 100 % before NVDEC does.

Bug Incidence

Always, under sustained multi-stream NVDEC load.

nvidia-bug-report.log.gz

Not attached here because it contains host details. We can send it privately on request.

More Info

Any of these would remove the limit, in order of preference:

1. Route interrupt sources over the allocated vectors, for example one per engine or runlist (NVDEC, copy engines, graphics/compute non-stall), so that irqbalance or the administrator can put them on different CPUs.
2. Make nvidia_isr() safe to run concurrently per vector: replace the per-device msix_isr_lock with finer-grained locking, so the vectors can be serviced in parallel.
3. Move less work into hard-IRQ context. 53-56 µs per interrupt is long. If the non-urgent part ran in the threaded handler (nvidia_isr_msix_kthread_bh) or in a workqueue not bound to the IRQ's CPU, it could run on other cores.
4. Interrupt moderation for completion (non-stall) interrupts: a module parameter to coalesce them by time or count, which accepts a small latency cost for a lower rate.

If an existing registry key or module parameter already does any of this, please tell us which GPUs and driver branches support it. Today /proc/driver/nvidia/params shows EnableMSI: 1 and empty RegistryDwords.

Our workarounds: pin vector 0 to a CPU outside the application's set (not always possible in a 28-vCPU VM), keep the stream count per GPU below the measured limit, and spread decode over more GPUs (one interrupt CPU per GPU).