Pi 5 / 6.18.39: rpi_hevc_dec VIDIOC_STREAMOFF blocks forever (hevc_d_h265_stop) after a decode request stalls on a missing reference picture — process unkillable, decoder lost until reboot

[b14ckyy]

Describe the bug

On a Raspberry Pi 5, when the stateless HEVC decoder (rpi_hevc_dec, /dev/video19, used through GStreamer's v4l2slh265dec) is asked to decode a P-frame whose reference picture is missing, the hardware never completes that request. That part may be expected (garbage in). The bug is what happens next: VIDIOC_STREAMOFF on the device then blocks forever in hevc_d_h265_stop(). The calling process goes to D state, cannot be killed, and every later open of the decoder blocks as well — the HEVC decoder is gone until a reboot.

In practice any UDP packet loss on a live HEVC stream (RTSP → v4l2slh265dec) triggers it, because the picture after the loss references a frame that never arrived. A GStreamer client sees Decoding frame N took too long from gstv4l2codech265dec.c, tears the pipeline down, and hangs in the teardown.

To reproduce

Deterministic, no network involved:

1. Take any HEVC Annex-B elementary stream (I used a 1280×720p60 Main-profile stream from an NVENC encoder, P-frames only; an x265 testsrc2 clip behaves the same).
2. Remove one P-frame (one VCL NAL of type TRAIL_R plus its prefix SEI) from the file, e.g. the first P-frame after the IDR.
3. Decode it on the hardware:

   gst-launch-1.0 filesrc location=gap.hevc ! h265parse ! decodebin3 ! videoconvert ! video/x-raw,format=BGRx ! fakesink sync=false
   

4. After ~1 s: v4l2slh265dec0: Decoding frame 1 took too long (gstv4l2decoder.c: Request 31 took too long.). gst-launch tries to shut the pipeline down and never returns; ps shows it in D state. After 120 s the hung-task detector fires (trace below). v4l2-ctl --list-devices and any further decoder open also block. Only a reboot recovers.

The same file with the P-frame present decodes fine at full speed, so the hardware and the driver are otherwise healthy.

Expected behaviour

VIDIOC_STREAMOFF must not block indefinitely. If the hardware does not finish a decode request, the driver should time out, reset the block and fail the request, so that a userspace client can recover (drop until the next IDR, reopen the device) without a reboot. Userspace cannot guarantee gap-free bitstreams on a lossy link.

Kernel trace

INFO: task gst-launch-1.0:14570 blocked for more than 120 seconds.
      Not tainted 6.18.39+rpt-rpi-2712 #1 Debian 1:6.18.39-1+rpt1
task:gst-launch-1.0  state:D stack:0     pid:14570 tgid:14570 ppid:14568  task_flags:0x400000 flags:0x00000009
Call trace:
 __switch_to+0xf0/0x1c8 (T)
 __schedule+0x3c8/0xc18
 schedule+0x3c/0xf0
 hevc_d_h265_stop+0xe4/0x120 [rpi_hevc_dec]
 hevc_d_stop_streaming+0x78/0x90 [rpi_hevc_dec]
 __vb2_queue_cancel+0x34/0x2e0 [videobuf2_common]
 vb2_core_streamoff+0x2c/0xd8 [videobuf2_common]
 vb2_streamoff+0x20/0x80 [videobuf2_v4l2]
 v4l2_m2m_streamoff+0x50/0x160 [v4l2_mem2mem]
 v4l2_m2m_ioctl_streamoff+0x20/0x40 [v4l2_mem2mem]
 v4l_streamoff+0x2c/0x40 [videodev]
 __video_do_ioctl+0x354/0x410 [videodev]
 video_usercopy+0x2c8/0x760 [videodev]
 video_ioctl2+0x20/0x40 [videodev]
 v4l2_ioctl+0x48/0x78 [videodev]
 __arm64_sys_ioctl+0xb4/0x120
 invoke_syscall+0x4c/0x100
 el0_svc_common.constprop.0+0x48/0xf0
 do_el0_svc+0x24/0x38
 el0_svc+0x38/0x120
 el0t_64_sync_handler+0xa0/0xe8
 el0t_64_sync+0x198/0x1a0

The same trace appears when the process is killed with SIGKILL while a decode is stalled (the v4l2_release path instead of the ioctl).

System

• Raspberry Pi 5 Model B Rev 1.0, Raspberry Pi OS (Debian 13 / trixie), 64-bit
• Linux 6.18.39+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.18.39-1+rpt1 (2026-07-29) aarch64
• linux-image-rpi-2712 1:6.18.39-1+rpt1, raspi-firmware 1:1.20260521-3, rpi-eeprom 28.30-1
• GStreamer 1.26.2 (gstreamer1.0-plugins-bad 1.26.2-3+rpt3+deb13u2), v4l2slh265dec
• Device registered as rpi-hevc-dec 1000800000.codec: Device registered as /dev/video19

Possibly related: #7537 (Pi 4B, v4l2slh265dec pipeline unkillable in D state after a seek). This one is on the Pi 5 and needs no seek, just a missing reference picture.

[popcornmix]

Might be worth a test with #7583.
e.g. using rpi-update. Read the rpi-update caveats first.

[b14ckyy]

Tested with rpi-update (6.18.48-v8-16k+, which contains #7583) on the same Pi 5: the file with the missing P-frame now fails cleanly with "Driver did not accept the bitstream parameters", the pipeline tears down, and the decoder is immediately usable again — no D state, no hung task. Same result on a second run, and an intact stream decodes as before. That fixes the lockup for me, thanks.

One thought for later: over a lossy link (RTSP/UDP from an NVENC encoder in my case) a missing reference picture is a regular event, and a software decoder just conceals it and carries on. It might be worth letting the hardware path tolerate that too instead of failing the stream — for now I drop pictures until the next keyframe on the client side. No more deadlocks is the important part.

[popcornmix]

The hardware decoder has no "concealment" built in.
All that could theoretically be done is to use the wrong reference frame (either a random one from DPB or a gray frame) to predict from but neither scheme would look good.

Ping @jc_kynesim