No video displayed with gpu video output on Raspberry Pi

[deltabeard]

mpv version and platform

Git master (a5f53da)
Raspberry Pi 3 - Buildroot image with patches:

0004-rpi-compile-fix.patch.txt
0003-gpu.patch.txt (Recently merged)
0002-fix-powerpc64-altivec.patch.txt
0001-wscript-don-t-check-for-fork-on-POSIX-checks.patch.txt

Issue additionally occurs on 0.27.0.

Reproduction steps

Play any h264 encoded video on the Raspberry Pi with the gpu video output driver enabled.
mpv --vo=gpu FILE
This is using the Raspberry Pi userland drivers, not vc4.

Expected behavior

To be able to see the video that is being played. The rpi driver does not have this issue.

Actual behavior

The subtitles and the OSD are both displayed on the screen, but the video itself is not displayed.

Frame drops still occur when subtitles and OSD is displayed, just like with the rpi video output driver.

Log file

log

Sample files

Any h264 encoded video file. Other video files were not tested, but may be able to reproduce the issue.

[ghost]

Did you try what I told you to try?

[deltabeard]

Yes, I did that yesterday and made a comment here.

[ghost]

Then take it up with the firmware maintainers. From what I can tell we're actually decode some video and are giving it the mmal video output video, so it's expected to work.

[deltabeard]

Looking at hwdec_rpi.c and vo_rpi.c, I think mpv does the same (or at least there is some resemblance) in the rpi video out driver too.

The layer used to configurable, but with gpu it seems that the layer is set automatically here. The default layer is 0, but with rpi the default layer was -10. Unsure why the magic number -10 was used.

I will have a more thorough look later.

[deltabeard]

Setting the layer to -10 here didn't make any difference.

I will continue to use the rpi video output driver in the meantime.

Edit: I deleted a comment after this one since it meant for a different issue. Apologies.

[deltabeard]

Setting hwdec to none works, the video is displayed on the screen. So the issue only occurs when using --vo=gpu --hwdec=mmal.

[cyph84]

@wm4 Are you positive its a firmware issue? Using 7428272 with the mpv commandline:
./mpv -v --hwdec=mmal --vo=gpu ~/test.mp4

Listing DISPMANX layers using
/opt/vc/bin/vcgencmd dispmanx_list
I see:
display:2 format:RGBA32 transform:20000 layer:0 src:0,0,853,480 dst:213,120,853,480 cost:563 lbm:0

In comparison to mpv 2.4.2 --hwdec=rpi --vo=rpi i get this:

display:2 format:YUV_UV transform:0 layer:-9 src:0,0,720,480 dst:0,0,1920,1080 cost:1216 lbm:12288 display:2 format:RGBA32 transform:20000 layer:1 src:0,0,1920,1080 dst:0,0,1920,1080 cost:1156 lbm:0

These are done with a blanked console, so you don't see the console layer showing up.

[itshaadi]

I was able to use GPU with --vo=gl --hwdec=rpi on mpv-0.26.0, ffmpeg-master. although SDL example doesn't work.
platform: RPI3-Raspbian-lite (stretch) with openbox.

[afflerbach]

I have the same (?) problem on an old Model B Rev 2 running Raspbian Buster.

With --hwdec=mmal-copy instead of --hwdec=mmal I can actually see the video, but this leads to lots of frame drops and A/V go completely out of sync.

$ mpv --hwdec=mmal video.mkv
…
Using hardware decoding (mmal).
AO: [alsa] 48000Hz stereo 2ch s16
VO: [gpu] 854x480 mmal
[vo/gpu] Using HW-overlay mode. No GL filtering is performed on the video!
AV: 00:00:03 / 00:01:24 (4%) A-V:  0.000 Dropped: 1

vs.

$ mpv --hwdec=mmal-copy video.mkv
…
Using hardware decoding (mmal-copy).
AO: [alsa] 48000Hz stereo 2ch s16
VO: [gpu] 854x480 yuv420p
AV: 00:00:03 / 00:01:24 (4%) A-V:  0.077 Dropped: 72

---

By the way: It works with omxplayer.

$ ldd ./omxplayer.bin | grep -iE gl\|mmal
        libbrcmGLESv2.so => /opt/vc/lib/libbrcmGLESv2.so (0xb6ed9000)
        libbrcmEGL.so => /opt/vc/lib/libbrcmEGL.so (0xb6ea0000)
        libmmal_core.so => /opt/vc/lib/libmmal_core.so (0xb4de7000)
        libmmal_util.so => /opt/vc/lib/libmmal_util.so (0xb4dc7000)
        libmmal_vc_client.so => /opt/vc/lib/libmmal_vc_client.so (0xb4dac000)

$ ldd ./mpv | grep -iE gl\|mmal
        libbrcmEGL.so => /opt/vc/lib/libbrcmEGL.so (0xb5b4d000)
        libbrcmGLESv2.so => /opt/vc/lib/libbrcmGLESv2.so (0xb5b28000)
        libmmal.so => /opt/vc/lib/libmmal.so (0xb594d000)
        libmmal_core.so => /opt/vc/lib/libmmal_core.so (0xb592f000)
        libmmal_util.so => /opt/vc/lib/libmmal_util.so (0xb590f000)
        libmmal_vc_client.so => /opt/vc/lib/libmmal_vc_client.so (0xb58f4000)
        libEGL.so.1 => /usr/lib/arm-linux-gnueabihf/libEGL.so.1 (0xb517d000)
        libGL.so.1 => /usr/lib/arm-linux-gnueabihf/libGL.so.1 (0xb4f6a000)
        libmmal_components.so => /opt/vc/lib/libmmal_components.so (0xb4af5000)
        libGLX.so.0 => /usr/lib/arm-linux-gnueabihf/libGLX.so.0 (0xb464c000)
        libGLdispatch.so.0 => /usr/lib/arm-linux-gnueabihf/libGLdispatch.so.0 (0xb45bb000)
        libwayland-egl.so.1 => /usr/lib/arm-linux-gnueabihf/libwayland-egl.so.1 (0xb4342000)

[Dudemanguy]

This might work now with the meson build (there were lots of weird things in waf), so I'll go ahead and close this one. Feel free to reopen if there is still an issue.

[chewi]

Sorry for the necrobump, but did this issue persist? I'm seeing it with 0.36. This is on Gentoo, where I'm trying to sort out the firmware/userland mess for our users, but it's turning out to be a lot harder than I expected. mmal-copy works but could be better. omxplayer works but that's unmaintained and locked into ffmpeg-4.

[philipl]

Are you using the recommended ffmpeg from the wiki?

[deltabeard]

Sorry for the necrobump, but did this issue persist? I'm seeing it with 0.36. This is on Gentoo, where I'm trying to sort of the firmware/userland mess for our users, but it's turning out to be a lot harder than I expected. mmal-copy works but could be better. omxplayer works but that's unmaintained and locked into ffmpeg-4.

I don't remember this issue being fixed. I think a lot has changed in the Raspberry Pi userland, mpv, ffmpeg, and Kernel, since I first opened this issue. This would need to be retested with and without VC4 on Raspberry Pi 3. I haven't used a Pi for a while, so it will be sometime before I can test this myself.

[chewi]

Are you using the recommended ffmpeg from the wiki?

Ah, I've missed that. That's unfortunate, I thought everything was upstreamed now. I'll see if it helps, thanks.

I am aware that a lot has changed, with a general shift towards full KMS. I trust that will work, though I haven't tried that yet, but I'm trying to also support Fake KMS for older Pi models that might struggle performance-wise.

[chewi]

I made a giant patch using jc-kynesim's dev/6.0/rpi_import_1 branch and it worked using --vo=gpu --hwdec=v4l2m2m ! Thank you very much for the pointer. I'm not sure what to do now though, as the Gentoo ffmpeg maintainers won't want to carry this huge patch.

I get the impression that this wasn't necessary in the past with --hwdec=mmal though. It isn't clear what broke it.

[philipl]

I'm glad it's working for you. Out of curiosity, can you attach a log so I can see what it's actually doing with that hwdec?

[philipl]

I made a giant patch using jc-kynesim's dev/6.0/rpi_import_1 branch and it worked using --vo=gpu --hwdec=v4l2m2m ! Thank you very much for the pointer. I'm not sure what to do now though, as the Gentoo ffmpeg maintainers won't want to carry this huge patch.

The most useful, but hardest, thing to do is try and advocate within the rpi community for upstreaming all this stuff.

[chewi]

I realised the other fork had a much smaller diff, but unfortunately it segfaults on 2 of my 3 test videos and the other video isn't quite as smooth. I realise I probably don't need all of the first fork, but picking out the bits I do need seems tedious. I'll add my weight to upstreaming where I can, but I'm not sure where that's most effective. Sounds like it needs to be on the upstream ffmpeg side?

Here's the log with --vo=gpu --hwdec=mmal while using the first ffmpeg fork:

[cplayer] Command line options: '--ao=null' '--vo=gpu' '--hwdec=mmal' '-v' '/home/chewi/test.mp4'
[cplayer] mpv 0.36.0 Copyright © 2000-2023 mpv/MPlayer/mplayer2 projects
[cplayer]  built on UNKNOWN
[cplayer] FFmpeg version: 6.0
[cplayer] FFmpeg library versions:
[cplayer]    libavutil       58.2.100
[cplayer]    libavcodec      60.3.100
[cplayer]    libavformat     60.3.100
[cplayer]    libswscale      7.1.100
[cplayer]    libavfilter     9.3.100
[cplayer]    libswresample   4.10.100
[cplayer] 
[cplayer] Configuration: -Db_pch=false -Dwerror=false -Dcplayer=true -Dlibmpv=true -Dtests=false -Dhtml-build=enabled -Dmanpage-build=enabled -Dpdf-build=disabled -Dbuild-date=false -Dlibarchive=disabled -Dlibbluray=disabled -Dcdda=disabled -Dcplugins=enabled -Ddvbin=disabled -Ddvdnav=disabled -Dsdl2-gamepad=disabled -Diconv=enabled -Djavascript=disabled -Dlibavdevice=enabled -Dlcms2=disabled -Dlua=luajit -Drubberband=disabled -Dsdl2=disabled -Duchardet=enabled -Dvapoursynth=disabled -Dzimg=disabled -Dzlib=enabled -Dalsa=enabled -Dcoreaudio=disabled -Djack=disabled -Dopenal=disabled -Dpipewire=disabled -Dpulse=disabled -Dsdl2-audio=disabled -Dsndio=disabled -Dx11=disabled -Dcocoa=disabled -Ddrm=enabled -Dgbm=enabled -Djpeg=disabled -Dcaca=disabled -Dlibplacebo=disabled -Drpi-mmal=enabled -Dsdl2-video=disabled -Dsixel=disabled -Dwayland=disabled -Dxv=disabled -Dgl=enabled -Degl=enabled -Degl-x11=disabled -Degl-drm=enabled -Degl-wayland=disabled -Dplain-gl=enabled -Dgl-x11=disabled -Dgl-cocoa=disabled -Drpi=enabled -Dvulkan=disabled -Dshaderc=disabled -Dcuda-hwaccel=disabled -Dcuda-interop=disabled -Dvaapi=disabled -Dvaapi-x11=disabled -Dvaapi-x-egl=disabled -Dvaapi-drm=disabled -Dvaapi-wayland=disabled -Dvdpau=disabled -Dvdpau-gl-x11=disabled -Dvideotoolbox-gl=disabled -Dprefix=/usr -Dlibdir=lib -Dlocalstatedir=/var/lib -Dsysconfdir=/etc -Dbuildtype=plain -Dwrap_mode=nodownload -Dbuild.pkg_config_path=/usr/share/pkgconfig -Dpkg_config_path=/var/tmp/portage/media-video/mpv-0.36.0-r1/temp/python3.11/pkgconfig:/var/tmp/portage/media-video/mpv-0.36.0-r1/temp/luajit/pkgconfig:/usr/share/pkgconfig --cross-file=/var/tmp/portage/media-video/mpv-0.36.0-r1/temp/meson.armv7a-unknown-linux-gnueabihf.arm.ini --native-file=/var/tmp/portage/media-video/mpv-0.36.0-r1/temp/meson.x86_64-pc-linux-gnu.arm.ini
[cplayer] List of enabled features: alsa av-channel-layout avif_muxer cplugins dmabuf-interop-gl drm drm-is-kms egl egl-drm egl-helpers ffmpeg gbm gl glibc-thread-name glob glob-posix gpl html-build iconv jpegxl libass libavdevice libdl libm librt linux-fstatfs luajit manpage-build noexecstack posix posix_shm rpi rpi-mmal stdatomic threads uchardet vector vt.h zlib
[cplayer] Built with NDEBUG.
[cplayer] Reading config file /etc/mpv/encoding-profiles.conf
[file] Opening /etc/mpv/encoding-profiles.conf
[cplayer] Applying profile 'default'...
[cplayer] Setting option 'ao' = 'null' (flags = 8)
[cplayer] Setting option 'vo' = 'gpu' (flags = 8)
[cplayer] Setting option 'hwdec' = 'mmal' (flags = 8)
[cplayer] Setting option 'v' = '' (flags = 8)
[cplayer] Waiting for scripts...
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.13 (SIMPLE) HarfBuzz-ng 8.0.1 (COMPLEX)
[osd/libass] Setting up fonts...
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: user-data/osc/visibility="auto" -> 1
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: user-data/osc/margins={"l":0,"r":0,"t":0,"b":0} -> 1
[cplayer] Done loading scripts.
[cplayer] Running hook: ytdl_hook/on_load
[ytdl_hook] ytdl:// hook 
[ytdl_hook] not a ytdl:// url 
[cplayer] Set property: shared-script-properties -> 1
[file] Opening /home/chewi/test.mp4
[cplayer] Set property: user-data/osc/margins={"l":0,"r":0,"t":0,"b":0} -> 1
[demux] Trying demuxers for level=normal.
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: user-data/osc/margins={"l":0,"r":0,"t":0,"b":0} -> 1
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.13 (SIMPLE) HarfBuzz-ng 8.0.1 (COMPLEX)
[osd/libass] Setting up fonts...
[lavf] Found 'mov,mp4,m4a,3gp,3g2,mj2' at score=100 size=2048.
[file] stream level seek from 131072 to 126507264
[demux] Detected file format: mov,mp4,m4a,3gp,3g2,mj2 (libavformat)
[cplayer] Opening done: /home/chewi/test.mp4
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[find_files] Loading external files in /home/chewi/
[cplayer] Running hook: ytdl_hook/on_preloaded
[lavf] select track 0
[lavf] select track 1
[cplayer]  (+) Video --vid=1 (*) (h264 854x480 29.970fps)
[cplayer]  (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
[vo/gpu] Probing for best GPU context.
[vo/gpu/opengl] Initializing GPU context 'rpi'
[vo/gpu/opengl] EGL_VERSION=1.4
[vo/gpu/opengl] EGL_VENDOR=Mesa Project
[vo/gpu/opengl] EGL_CLIENT_APIS=OpenGL OpenGL_ES 
[vo/gpu/opengl] Trying to create Desktop OpenGL context.
[vo/gpu/opengl] Could not choose EGLConfig for Desktop OpenGL!
[vo/gpu/opengl] Trying to create GLES 2.x + context.
[vo/gpu/opengl] Could not choose EGLConfig for GLES 2.x +!
[vo/gpu/opengl] Could not create a GL context.
[vo/gpu/opengl] Initializing GPU context 'drm'
[vo/gpu/drm] Picked DRM card 0, primary node /dev/dri/card0 as the default.
[vo/gpu/drm] Driver: vc4 0.0.0 (20140616)
[vo/gpu/drm] Connector 89 currently connected to encoder 88
[vo/gpu/drm] Selected Encoder 88 with CRTC 87
[vo/gpu/drm] Could not find any preferred mode. Picking the first mode.
[vo/gpu/drm] Selected mode: 1920x1080 (1920x1080@60.00Hz)
[vo/gpu/drm] DRM Atomic support found
[vo/gpu/drm] Using primary plane 31 as draw plane
[vo/gpu/drm] Using overlay plane 38 as drmprime plane
[vo/gpu] GBM_FORMAT_ARGB8888 supported by draw plane.
[vo/gpu] Supported modifier: 0x0
[vo/gpu] Supported modifier: 0x700000000000001
[vo/gpu] Creating GBM device
[vo/gpu] Initializing GBM surface (1920 x 1080)
[vo/gpu/opengl] Initializing EGL
[vo/gpu/opengl] EGL_VERSION=1.4
[vo/gpu/opengl] EGL_VENDOR=Mesa Project
[vo/gpu/opengl] EGL_CLIENT_APIS=OpenGL OpenGL_ES 
[vo/gpu/opengl] Trying to create Desktop OpenGL context.
[vo/gpu/opengl] Attempting to find EGLConfig matching GBM_FORMAT_ARGB8888
[vo/gpu/opengl] Found matching EGLConfig for GBM_FORMAT_ARGB8888
[vo/gpu/opengl] Initializing EGL surface
[vo/gpu] GL_VERSION='2.1 Mesa 23.1.8'
[vo/gpu] Detected desktop OpenGL 2.1.
[vo/gpu] GL_VENDOR='Broadcom'
[vo/gpu] GL_RENDERER='VC4 V3D 2.1'
[vo/gpu] GL_SHADING_LANGUAGE_VERSION='1.20'
[vo/gpu] Loaded extension GL_ARB_map_buffer_range.
[vo/gpu] Loaded extension GL_ARB_framebuffer_object.
[vo/gpu] Loaded extension GL_ARB_vertex_array_object.
[vo/gpu] Loaded extension GL_ARB_sync.
[vo/gpu] Loaded extension GL_ARB_invalidate_subdata.
[vo/gpu] Loaded extension GL_ARB_get_program_binary.
[vo/gpu] Loaded extension GL_ARB_buffer_storage.
[vo/gpu] Loaded extension GL_ARB_debug_output.
[vo/gpu] Loaded extension GL_KHR_debug.
[vo/gpu/opengl] Preparing framebuffer
[vo/gpu/opengl] GBM surface using modifier 0x700000000000001
[vo/gpu/drm] crtc is not VRR capable
[vo/gpu/drm] Monitor pixel aspect: 1
[vo/gpu/opengl] Opening render node "/dev/dri/renderD128"
[vo/gpu/opengl] 16 bit UNORM textures not available.
[vo/gpu] Disabling HDR peak computation (one or more of the following is not supported: compute shaders=0, SSBO=0).
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vo/gpu] Assuming 60.000000 FPS for display sync.
[vd] Container reported FPS: 29.970000
[vd] Codec list:
[vd]     h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd]     h264_v4l2m2m (h264) - V4L2 mem2mem H.264 decoder wrapper
[vd]     h264_mmal (h264) - h264 (mmal)
[vd] Opening decoder h264
[vd] Looking at hwdec h264_mmal-mmal...
[vo/gpu] Loading hwdec drivers for format: 'mmal'
[vo/gpu] Loading hwdec driver 'rpi-overlay'
[vd] Trying hardware decoding via h264_mmal-mmal.
[vd] Using underlying hw-decoder 'h264_mmal'
[vd] Pixel formats supported by decoder: mmal yuv420p
[vd] Codec profile: unknown (0xffffff9d)
[vd] Requesting pixfmt 'mmal' from decoder.
[vd] Selected codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
[vf] User filter list:
[vf]   (empty)
[ad] Codec list:
[ad]     aac - AAC (Advanced Audio Coding)
[ad]     aac_fixed (aac) - AAC (Advanced Audio Coding)
[ad] Opening decoder aac
[ad] Requesting 1 threads for decoding.
[ad] Selected codec: aac (AAC (Advanced Audio Coding))
[af] User filter list:
[af]   (empty)
[cplayer] Starting playback...
[file] stream level seek from 126580726 to 79
[af] [in] 48000Hz stereo 2ch floatp
[af] [userspeed] 48000Hz stereo 2ch floatp
[af] [userspeed] (disabled)
[af] [convert] 48000Hz stereo 2ch floatp
[vd] Using hardware decoding (mmal).
[vd] Decoder format: 854x480 [0:1] mmal auto/auto/auto/auto/auto CL=unknown
[vf] [in] 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] (disabled)
[vf] [autorotate] 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [autorotate] (disabled)
[vf] [convert] 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [convert] (disabled)
[vf] [out] 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[ao] Trying audio driver 'null'
[ao/null] requested format: 48000 Hz, stereo channels, floatp
[ao/null] Channel layouts:
[ao/null]  - anything
[ao/null] result: stereo
[ao/null] device buffer: 9472 samples.
[ao/null] using soft-buffer of 9600 samples.
[cplayer] AO: [null] 48000Hz stereo 2ch floatp
[cplayer] AO: Description: Null audio output
[af] [convert] (disabled)
[af] [out] 48000Hz stereo 2ch floatp
[cplayer] VO: [gpu] 854x480 mmal
[cplayer] VO: Description: Shader-based GPU Renderer
[vo/gpu] reconfig to 854x480 mmal bt.601/bt.601-525/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 854x480 (1:1)
[vo/gpu] Video display: (0, 0) 854x480 -> (0, 0) 1920x1079
[vo/gpu] Video scale: 2.248244/2.247917
[vo/gpu] OSD borders: l=0 t=0 r=0 b=1
[vo/gpu] Video borders: l=0 t=0 r=0 b=1
[vo/gpu] Reported display depth: 8
[vo/gpu] Using HW-overlay mode. No GL filtering is performed on the video!
[vo/gpu] No advanced processing required. Enabling dumb mode.
[cplayer] first video frame after restart shown
[vo/gpu/opengl] GBM surface using modifier 0x700000000000001
[cplayer] audio ready
[cplayer] delaying audio start 0.000000 vs. 0.000000, diff=0.000000
[cplayer] playback restart complete @ 0.000000, audio=ready, video=playing
[cplayer] starting audio playback
[vo/gpu/opengl] GBM surface using modifier 0x700000000000001
[ao/null] starting AO
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: user-data/osc/margins={"l":0,"r":0,"t":0,"b":0} -> 1
[vo/gpu/opengl] GBM surface using modifier 0x700000000000001
[lavf] EOF reached.
[cplayer] EOF code: 5  
[cplayer] finished playback, success (reason 3)
[cplayer] 
[cplayer] Exiting... (Quit)
[cplayer] Set property: shared-script-properties -> 1

[chewi]

False alarm with the segfaults on jernejsk's fork. I had used the v4l2-drmprime-n6.0 branch. The v4l2-request-n6.0 branch works instead. I gather you can combine these though. This is what Arch does.

I went on to test the situation under full KMS with jc-kynesim's fork. It works with these arguments.

--vo=gpu --hwdec=v4l2m2m  --drm-draw-plane=overlay --drm-drmprime-video-plane=primary

Given that we need these patches for full KMS, there seems little point in --hwdec=mmal now, so don't feel the need to spend time investigating that.

My testing has only been 32-bit so far. I'll test 64-bit soon.