Missing decoders for Media Foundation

[dariox86]

Game information
Unravel (1225560)
Blair WItch (1092660)

Distribution name and version where applicable
Parabola GNU/Linux-libre x86-64 (rolling release)

Flatpak info
Flatpak 1.9.2
default
host

Problem description
Recent work in Proton made many cutscenes working in games using Media Foundation. In Steam for Flatpak with Proton-GE 5.21-GE-1, playback for such cutscenes is not working. When cutscenes are supposed to be shown, a color bar is shown instead. If the videos in the game folder are converted to a royalty free format such as VP9, cutscenes work. For example as a workaround for Unravel (1225560), open $HOME/.var/app/com.valvesoftware.Steam/data/Steam/steamapps/common/Unravel/D3D11/V and convert all the video files to VP9 with a line such as: $ ffmpeg -i [input file] -c:v libvpx-vp9 -crf 30 -b:v 0 [output file].webm. Then delete the original files and the .webm extension from the converted files and launch the game with Proton-GE 5.21-GE-1.
I am unsure whether the issue is related to Flatpak Steam or Flatpak Proton-GE.
I have attached two logs. One was taken with the unmodified game and the other one with the video files converted to VP9.

Does this issue reproduce with native Steam
This issue is not reproducible with native Steam. In native Steam cutscenes for these games work out of the box with Proton-GE 5.21-GE-1.

logs.zip

[dariox86]

For the sake of clarity - cutscenes for Unravel (1225560) are stored as DivX/AAC files in a game folder. With the latest Proton-GE cutscene playback works in native Steam but not in Flatpak Steam. The same goes for Blair Witch (1092660). Replacing these video files with a copy reencoded to a royalty free format such as VP8 or VP9 with the same name and extension as the originals results in working cutscenes in Flatpak Steam. No such conversion is needed in native Steam.

[gasinvein]

Do you have org.freedesktop.Platform.ffmpeg-full//20.08 and org.freedesktop.Platform.ffmpeg_full.i386//20.08 installed?

[dariox86]

I had already tried. Same problem. It is easy to reproduce if you happen to have any Media Foundation-based game.

[gasinvein]

Well, I have only one wmf-based game - Thronebreaker from GOG. It doesn't work with current Proton, but works with Proton-Exp (both flatpak). Didn't test with Proton-GE.
In any case, please launch Steam via flatpak run --env=GST_DEBUG=4 com.valvesoftware.Steam and attach it's output when running the game. Maybe we'll see then what's wrong with gstreamer.

[dariox86]

Attached is the relevant log. This is the placeholder I see instead of the actual video.

Unravel-1225560-missing-cutscenes.log

[gasinvein]

Just tested Blair WItch. Videos work for me with Proton-GE, but don't with Proton and Proton-Exp. Strangely enough, I don't see the color bar, just black screen.
@nanonyme Proton-GE flatpak has bundled gstreamer, but still uses the same ffmpeg from runtime. Is it possible that freedesktop-sdk's gstreamer is missing something needed to decode media?

[dariox86]

@gasinvein, in Blair Witch (1092660) with Proton 5.13-4 I only see a black screen, but with Proton-GE 5.21-GE-1 I see the color bar.

[dariox86]

@gasinvein, do you see the cutscenes in Blair Witch with Proton-GE 5.21-GE-1 without reencoding the video files?

[gasinvein]

@dariox86 Yes, didn't touch game files. For clarity, I've used current flatpak build of Proton-GE.

[dariox86]

Weird... this is the list of packages I have installed. Am I missing something?

Name                                      Application ID                                        Version     Branch   Origin        Installation
Steam                                     com.valvesoftware.Steam                               1.0.0.68    stable   flathub       user
Proton (community build)                  com.valvesoftware.Steam.CompatibilityTool.Proton      5.13-4      stable   flathub       user
Proton-GE (community build)               com.valvesoftware.Steam.CompatibilityTool.Proton-GE   5.21-GE-1   stable   flathub       user
protontricks                              com.valvesoftware.Steam.Utility.protontricks          1.4.3       stable   flathub       user
Lutris                                    net.lutris.Lutris                                     0.5.8.2     beta     flathub-beta  user
Freedesktop Platform                      org.freedesktop.Platform                              19.08.12    19.08    flathub       user
Freedesktop Platform                      org.freedesktop.Platform                              20.08.3     20.08    flathub       user
i386                                      org.freedesktop.Platform.Compat.i386                              20.08    flathub       user
default                                   org.freedesktop.Platform.GL.default                               19.08    flathub       user
default                                   org.freedesktop.Platform.GL.default                               20.08    flathub       user
default                                   org.freedesktop.Platform.GL32.default                             20.08    flathub       user
ffmpeg-full                               org.freedesktop.Platform.ffmpeg-full                              20.08    flathub       user
i386                                      org.freedesktop.Platform.ffmpeg_full.i386                         20.08    flathub       user
openh264                                  org.freedesktop.Platform.openh264                     2.1.0       2.0      flathub       user
GNOME Application Platform version 3.38   org.gnome.Platform                                                3.38     flathub       user
i386                                      org.gnome.Platform.Compat.i386                                    3.38     flathub       user
wxHexEditor                               org.wxhexeditor.wxHexEditor                           0.24        stable   flathub       user

[gasinvein]

No, everything seems in place. Weird indeed. Did you try clearing compatdata for the game? It also contains gstreamer cache.

[dariox86]

I deleted and reinstalled Steam for Flatpak multiple times, sometimes I have reinstalled the whole operating system.

[gasinvein]

@dariox86 Can you please try disabling shader cache in Steam settings and retry Blair Witch with Proton-GE flatpak?

[dariox86]

Unbelievable... it works! Re-enabling shader caching again I see the color bars!

[gasinvein]

Well, here are my findings (may be inaccurate):

• When shader cache is enabled, Steam Client sets STEAM_COMPAT_MEDIA_PATH environment variable, which, in turn, enables the proton media-converter gstreamer plugin (I guess it's no-op otherwise)
• media-converter doesn't work in our case and is able to only show color bars;
  I'm not sure if there is something wrong in the flatpak Proton build, since I don't completely understand how media-converter works (transcoded media, which it should use, is missing, but I don't know where it normally comes from)
• When media-converter is no-op, Wine uses gstreamer normally, playing videos with whatever codecs available to it;
  Videos in Blair Witch started to work for me in Proton-Exp when I replaced runtime ffmpeg with a local full-featured build

So, we have two issues here: media-converter doesn't work (and prevents fallback to other gstreamer plugins) and some codecs are missing from the runtime ffmpeg extension.

[dariox86]

I have read discussions about not including patent-encumbered decoders in org.freedesktop.Platform.ffmpeg-full, but I am not knowledgeable enough to understand the inner workings of Flatpak packages.

[dariox86]

Ironically Unravel (1225560) does not work with shader caching disabled. It goes ingame and keeps loading forever. I did not try running from a clean wineprefix though. @gasinvein, how do you use Proton-Experimental? Have you installed it from within Steam itself?

[tim77]

how do you use Proton-Experimental?

flatpak install com.valvesoftware.Steam.CompatibilityTool.Proton-Exp

[dariox86]

Thank you.

[dariox86]

Well, Unravel (1225560) does work with shader caching disabled, but only when launching with Proton Experimental 5.13-20210107. Moreover, the problem with red component color lacking is fixed with the experimental version.

[gasinvein]

@dariox86 We're still missing some ffmpeg codecs. Can you please again launch with GST_DEBUG=4 (or 3, 4 is too noisy) and post output when running Unravel with Proton-GE with shader cache disabled?

[dariox86]

@gasinvein, sure. I am not home at the moment. I will get to you later.

[dariox86]

Here it is. Working cutscenes with wrong colors.
Unravel-(1225560)-Proton-GE-5.21-GE-1-Shaders-Caching-Disabled.log

[gasinvein]

@dariox86 Ah, sorry, I've misread your previous comment. Actually I needed log from Proton Experimental, not GE.

[dariox86]

Unravel (1225560) launched with Proton Experimental 5.13-20210107 and shader caching enabled: as opposed to my previous test, color bars again. Same Proton version, shader caching disabled: cutscene playback working (with proper colors). Both tries in the log.
Unravel-(1225560)-Proton Experimental-5.13-20210107-1st-try-shader-caching-enabled-color-bar+2nd-try-shader-caching-disabled-working-cutscenes.log

[gasinvein]

@dariox86 thanks.

0:00:00.668972639  3506 0x7f8aec013520 WARN              video-info video-info.c:197:validate_colorimetry: Need to specify a color matrix when using YUV format (I420)
0:00:00.668986185  3506 0x7f8aec013520 WARN              video-info video-info.c:513:gst_video_info_from_caps: invalid colorimetry, using default

^ this might be the reason for the wrong colors. Also seems like a ffmpeg-related issue to me. Looks like we'll need to bundle it with steam flatpak (again).

[dariox86]

Well, in the same log the game was launched twice. The first test has wrong colors, but the second one has proper colors. From my limited knowledge, it appears to me that this same message is present even when colors are right in the second test. I don't know if the problem with the colors is related to Flatpak because it is the Proton version that makes a difference here. You know, even when cutscenes are not working and the color bar shows up instead, with Proton versions other than the latest experimental build the colors are wrong with the color bar, too.

[nanonyme]

@gasinvein so does it seem to you that we're still missing some codecs from ffmpeg-full?

[gasinvein]

@nanonyme I'm pretty sure we do, but not sure which ones. Looks like something around aac and/or quicktime (or divx).

[terzag]

Hello,
I have the same issue with Black Book Prologue (which is free on Steam): color bar instead of what I assume is a video playing. I've tried to install org.freedesktop.Platform.ffmpeg_full.i386 which was missing but no change. Using Proton-6.5-GE-3.

I've just tried disabling the shaders cache and it seems to fix the issue (at least in the main menu, I haven't tried starting a game yet).

[gasinvein]

@terzag try disabling shader cache.
Black Book Prologue shows the color bar for me both with official Proton and the community build when shader cache is enabled. With shader cache disabled, it seems to work properly with community build, but crashes with official Proton.

[terzag]

Yeah, I've tried to disable the shader cache and it fixed the issue, you might have seen my message before I tried and edited it. :)

[gasinvein]

@terzag Well, long story short, multimedia in Proton community build tends to work better with shader cache disabled.
@dariox86 Btw, videos in Blair Witch started to work for me with Proton community build (non-GE, non-experimental), as of 6.3-4. Is it the case for you as well?

[nanonyme]

@gasinvein that doesn't sound overly great if the assumption is moving mostly to official Proton once the sub-sandboxing becomes supported everywhere.

[gasinvein]

@nanonyme I still don't completely understand how their media-converter gstreamer plugin works, and not sure if it's expected to work for all games. For example, for me it works for Age of Empires II and Soulcalibur VI, but not for Blair Witch (in official Proton).

[dariox86]

@dariox86 Btw, videos in Blair Witch started to work for me with Proton community build (non-GE, non-experimental), as of 6.3-4. Is it the case for you as well?

@gasinvein, I am migrating my system to a new file system. It will take some time and I can't launch Steam at the moment. I will keep you updated.

[dariox86]

Sorry, I am very late. For the record, yes, now I think this is fixed. Even though I didn't test it thoroughly, Unravel (1225560) and Blair WItch (1092660) seem to work out of the box with cutscenes and shader pre-caching enabled under Proton-GE. At the moment, some restricted codecs only work on Proton-GE. Thank you!