setting hwdec to vaapi does not work

[dgel]

Overview Description:
I have a dell xps 9570 with integrated intel gpu and discrete nvidia 1050 gpu. I'm using the flatpak version of Celluloid, and when setting hwdec=auto and vo=gpu in the config for celluloid, it uses nvdec-copy, and when explicitly setting hwdec=vaapi, it falls back to the software decoder.

However, when running the mpv flatpak, I can freely choose between nvdec-copy and vaapi, both work as expected.

Steps to Reproduce:

1. use a system with intel / nvidia hybrid graphics
2. set vo=gpu and hwdec=vaapi in mpv config file
3. open a video file in flatpak celluloid
4. press i to inspect actual settings used

Actual Results:
software decoder is used

Expected Results:
vaapi decoder is used

Version: 0.23

Additional Information:

[dgel]

BTW, I just wanted to say thanks for this project. I've been using smplayer for over a decade, mostly for the mpv backend and access to settings for gpu acceleration, subtitle settings etc. Having found this project i realised most of those settings were actually provided by mpv, and Celluloid fits into my desktop much better than smplayer does, while exposing only those controls I regularly use. The others I can use the mpv keyboard shortcuts when needed

[gnome-mpv]

The problem is probably with hybrid graphics. VAAPI works fine on my machine, but I don't hybrid graphics to test with.

Would you be able to check if it works outside of Flatpak?

[dgel]

I could try to check outside of flatpak, is there a recommended way to install it on Fedora 36 outside of flatpak?

I'm not sure it's going to be relevant though, as I can run the mpv flatpak with the same settings and there the vaapi backend just works, as well as nvdec-copy. As it's also a flatpak, I don't think it's a result of my flatpak configuration, and I've checked in flatseal that my settings for both flatpaks is the same. My guess would be that some configuration in the Celluloid flatpak itself is different:

$ flatpak run io.mpv.Mpv --vo=gpu --hwdec=vaapi Big_Buck_Bunny_1080_10s_20MB.mp4
 (+) Video --vid=1 (*) (h264 1920x1080 60.000fps)
[ffmpeg] AVHWDeviceContext: Failed to query surface attributes: 20 (the requested function is not implemented).
[vo/gpu/vaapi-egl] failed to retrieve libavutil frame constraints
Using hardware decoding (vaapi).
VO: [gpu] 1920x1080 vaapi[nv12]
V: 00:00:01 / 00:00:10 (18%)

Exiting... (Quit)

$ flatpak run io.mpv.Mpv --vo=gpu --hwdec=nvdec-copy Big_Buck_Bunny_1080_10s_20MB.mp4
 (+) Video --vid=1 (*) (h264 1920x1080 60.000fps)
Using hardware decoding (nvdec-copy).
VO: [gpu] 1920x1080 nv12
V: 00:00:02 / 00:00:10 (29%)

Exiting... (Quit)

[dgel]

Actually, it's not due to flatpak, I ended up compiling celluloid locally, using the system's devel packages for dependencies, and I get the same result on here:

config used:

save-position-on-quit
vo=gpu
hwdec=vaapi
sub-auto=fuzzy
slang=en,eng,nl,nld

[D33M0N]

Same problem, but with amd solo GPU (vega64).
Best I could get was using hwdec=auto which turned to use "vaapi-copy", which is not ideal. Maybe because celluloid doesn't work or pass the vo=gpu to mpv, so I can't use vo=gpu + hwdec=vaapi (vaapi requires vo=gpu to function).

Even when I try to run it from command line with:
celluloid --mpv-vo=gpu --mpv-hwdec=vaapi videofile.mp4 it refuses to use it and reverts back to software decoder.

So, my assumption so far is that hwdec= gets passed on and works (when using any available hwdec options that do not require some specific vo= to work), HOWEVER Celluloid has problem with passing on vo=gpu to mpv, which is requirement for the hwdec=vaapi to work. ( https://mpv.io/manual/master/#video ), but I don't know how to test or debug this :-(

[ahumeniy]

Still have this issue on Thinkpad T480 (Intel HD Graphics 620, no discrete GPU) running Fedora 38. VLC has no issues playing video files at full framerate while Celluloid looks choppy.

[alexmercerind]

Wayland?

[ahumeniy]

Wayland?

Tested on Xorg. Still the same.

[zhmars]

• mpv 0.36
• celluloid 0.25
• GNOME 44.3 (Xorg)
• Intel HD Graphics 4400 (no hybrid graphics)

celluloid/src/celluloid-mpv.c

Lines 423 to 424 in c587612

	g_info("Forcing --vo=libmpv");
	mpv_set_option_string(priv->mpv_ctx, "vo", "libmpv");

It seems vo option is force to libmpv. Changing libmpv to gpu should make VAAPI work again, but the video will play on a external window. Not sure if it is a regression from upstream.

[alexmercerind]

The --wid does not work on Wayland: mpv-player/mpv#9654. Thus, --vo=gpu will always create a new window.

GL provided by GtkGLArea (from GTK) is used to draw video using render API of libmpv i.e. --vo=libmpv. The problem with this is that only *-copy hwdec(s) work due to GtkGLArea's GL version being low or missing certain features.
OR... it could be a bug within mpv which incorrectly handles initialization of hwdec(s).

[uralets]

Desktop Ubuntu 23.10 @ Intel 12100 here - hwdec=vaapi falls back to software rendering, while vaapi-copy works. With a huge performance penalty as far as CPU utilization goes.

mpv works with hwdec=vaapi as expected.

UPD
I'm on Mate DE, so using Xorg with the Picom compositor.
Problem appears to be in

[   0.012][v][libmpv_render] Loading hwdec driver 'vaapi'
[   0.012][v][libmpv_render/vaapi] VAAPI hwdec only works with OpenGL or Vulkan backends.
[   0.012][v][libmpv_render] Loading failed.             

Further investigation (googling) suggest (as per mpv-player/mpv#10424 (comment)) this might be because Celluloid uses GLX Opengl backend instead of EGL.