Fix Debian-12.packages

[koplo199]

Some packages/libraries listed in Debian-12.packages use their old name from Ubuntu-18.04.packages, and therefore do not exist anymore in Debian 12. This PR updates the packages/libraries names to their correct Debian Bookworm names.

Most of the time the package/library name only changed because of a version increment (e.g libvpx5 -> libvpx7).

[koplo199]

I also noticed something unexpected:
Taking the same libvpx example, on the latest Wine-GE-Proton8-17 doing an ldd lutris-GE-Proton8-17-x86_64/lib64/gstreamer-1.0/libgstvpx.so | grep libvpx.so outputs:

libvpx.so.6 => not found

Which does not match neither the original libvpx.so.5 nor the updated libvpx.so.7.
It seems this version matches the Debian Bullseye package.

I though newer Wine-GE releases were built on Debian Bookworm, why is it expecting to find a Debian Bullseye library?

Edit: libgstvpx.so comes from gstreamer1.0-plugins-good, and I verified by downloading it manually from the official Debian Bookworm deb or the deb-multimedia Bookworm deb and both correctly outputs libvpx.so.7 when running the above ldd command. It means the one embedded in Wine-GE is not from one of those two sources when, from my understanding, it should be.

Edit: Oh ok, it seems this library comes in fact from buildbot/runtime/extra/lib{32,64}.
By downloading from here I can reproduce the error:

libvpx.so.6 => not found

With any of the two Debian Bookworm deb:

libvpx.so.7 => /lib/x86_64-linux-gnu/libvpx.so.7 (0x00007f80e6200000)

Those extra libraries, apparently extracted from Proton, are not compatible with Debian Bookworm.

[GloriousEggroll]

I also noticed something unexpected: Taking the same libvpx example, on the latest Wine-GE-Proton8-17 doing an ldd lutris-GE-Proton8-17-x86_64/lib64/gstreamer-1.0/libgstvpx.so | grep libvpx.so outputs:

libvpx.so.6 => not found

Which does not match neither the original libvpx.so.5 nor the updated libvpx.so.7. It seems this version matches the Debian Bullseye package.

I though newer Wine-GE releases were built on Debian Bookworm, why is it expecting to find a Debian Bullseye library?

Edit: libgstvpx.so comes from gstreamer1.0-plugins-good, and I verified by downloading it manually from the official Debian Bookworm deb or the deb-multimedia Bookworm deb and both correctly outputs libvpx.so.7 when running the above ldd command. It means the one embedded in Wine-GE is not from one of those two sources when, from my understanding, it should be.

this is incorrect. As noted in the README here:

https://github.com/lutris/buildbot/blob/master/runtime/extra/README

Specifically:

The lib64 and lib32 folders are used for building purposes only. 

The contents of lib64 and lib32 folders are copied from steam-linux-runtime and steam-proton.

The contents ot steam-linux-runtime and steam-proton should be kept updated and imported from "Steam Linux Runtime" within any steam library and "Proton" within any Proton-GE build respectively.

The reason it's done this way is so that we can differentiate what we need to keep updated from the steam linux runtime and what we need from proton in order for the gstreamer libraries imported from proton to work properly.

The gstreamer build is copied from proton, therefore we needed the same libraries proton uses.

[koplo199]

Seems we're in sync, I've just seen that (I updated my comment at the same time you posted yours).
Doesn't resolve the issue though, those libraries are incompatible with the Debian 12 environment.

[GloriousEggroll]

Seems we're in sync, I just seen that (I updated my comment at the same time you posted yours). Doesn't resolve the issue though, those libraries are incompatible with the Debian 12 environment.

They don't need to be. lutris preloads the entire gstreamer folder (and it's lib64 and lib32 folder within the build), therefore all the relevant libraries are loaded as overrides, and therefore gstreamer finds all of the libraries it needs in order to function. The only time this does not happen is if you turn off the runtime in lutris, or use the build outside of lutris, which is not intended.

[GloriousEggroll]

To clarify, I'm not saying your wrong about the debian runtime folder/files, but I am saying you're wrong about libvpx.

All of the files in /runtime/extra/lib64 and /runtime/extra/lib32 are independant of the debian runtime. They exist specifically for supporting gstreamer functionality with gstreamer imported from proton.

[koplo199]

If I understand correctly:

1. At build time, gstreamer libraries from deb-multimedia are used.
2. At run time, the use of gstreamer libraries extracted from Proton are forced.
3. To accomodate that change, and because Proton gstreamer expects a very specific set of libraries that are not found on Debian 12, this set of libraries is also included in the builds.

If so:

• Why choosing Debian 12 as the runtime base? Why not using whatever platform Proton is build on¹?
• Or alternatively, why not building gstreamer with the same patchs² as Proton on Debian 12?

Both of these options would resolve libraries incompatibilities.

But currently, libraries built for very different platforms/environments are mixed, and I'm surprised it even works. This is pure luck, and relies on the other libraries between the two environments being not-so-different. It can break at any time and in the most subtle ways. I had the occasion once to debug an issue cause by different openssl versions being used at build time and run time, and I really don't recommend the experience.

Footnotes

1. Apparently Debian Bullseye, as the libraries hinted, and confirmed by the documentation: https://gitlab.steamos.cloud/steamrt/steamrt/-/blob/steamrt/sniper/README.md ↩

2. I don't know if gstreamer is patched directly. Maybe it is embedded because of compile time options not enabled in the official .deb. Or maybe it doesn't even have to be embedded, I can't tell. ↩

[koplo199]

I did some additional research to have a clearer view of the situation and came across one of your reddit post.

It states that gstreamer libraries are embedded because of some codecs not being available (no custom patch involved here apparently, which is nice). Is it still true for Debian 12 + deb-multimedia, does it still not provide those codecs?

If it does provides them, gstreamer could instead be embedded in the Lutris runtime by adding the concerned libraries to the Debian-12.packages. It will ensure a total compatibility with the rest of the libraries, save ~80 Mo per build, avoid all the issues listed in the above comment, be a lot easier to maintain, and the list goes on.

Edit: here is an experimental branch porting all the libraries listed in runtime/extra/lib{32,64}/gstreamer-1.0 to the Debian-12.packages file format

[GloriousEggroll]

It states that gstreamer libraries are embedded because of some codecs not being available (no custom patch involved here apparently, which is nice). Is it still true for Debian 12 + deb-multimedia, does it still not provide those codecs?

If it does provides them, gstreamer could instead be embedded in the Lutris runtime by adding the concerned libraries to the Debian-12.packages. It will ensure a total compatibility with the rest of the libraries, save ~80 Mo per build, avoid all the issues listed in the above comment, be a lot easier to maintain, and the list goes on.

Edit: here is an experimental branch porting all the libraries listed in runtime/extra/lib{32,64}/gstreamer-1.0 to the Debian-12.packages file format

We import the pre-built gstreamer libraries from proton-ge because we have full control of what codecs get enabled or disabled. Proton has some overrides for some codecs via the ongoing media foundation work, and enabling those codecs in gstreamer can cause breakage. If we import the ones from debian instead of the ones from proton, we have no control over what codecs/encoders/decoders etc are enabled or disabled via build flags. THIS is why we don't use debian's gstreamer.

To clarify what i mean by 'pre-built' -- in proton (and proton-ge), gstreamer is built from source, not imported from packages. Once those are built we copy them over to the wine-ge builds.

When proton is built, it uses the steam sniper sdk runtime, which is based on debian 11 but has several customized packages from valve directly. This is why the prebuilt gstreamer libraries require a few (like 2-3) libraries from that runtime.

The reason we install the gstreamer debian 12 packages as dependencies is so that wine picks up and detects the development files in order to enable gstreamer support during build time. Without doing so it has no headers to build gstreamer support from, thus gstreamer support gets disabled. Basically we trick the wine build into thinking there is gstreamer support coming from debian 12 via the header files from the devel packages, then override the expected libraries during the build with those from proton.

Hopefully that paints a clearer picture as to why we don't and can't use debian's gstreamer libraries instead of the ones from proton.

[koplo199]

Thanks for the detailed explanation, it seems the informations you gave in the reddit post are still relevant today.

We import the pre-built gstreamer libraries from proton-ge because we have full control of what codecs get enabled or disabled. [...] THIS is why we don't use debian's gstreamer.

And this is also why I asked: Does Debian 12 + deb-multimedia still not provide the required codecs by default? Best case scenario it does, and Debian's gstreamer could be used as is.

If we import the ones from debian instead of the ones from proton, we have no control over what codecs/encoders/decoders etc are enabled or disabled via build flags.

Assuming the worst case scenario where Debian's gstreamer does't provide the necessary codecs, we could still have full control over what's get enabled/disabled by building from source using apt-get source <package>, apt-get build-dep <package> and dpkg-buildpackage -rfakeroot -uc -b. It only requires a simple sed command to change the debian/rules file to enable/disable whatever codecs are needed.

[GloriousEggroll]

And this is also why I asked: Does Debian 12 + deb-multimedia still not provide the required codecs by default? Best case scenario it does, and Debian's gstreamer could be used as is.

It's not a matter of enabling options so much as it is disabling. Some options need to be disabled so that Valve's wine patched overrides can take effect.

And yes, we could rebuild debian's packages with the same build flags then provide a ppa for them, but it's more work for something that is already done (built in proton), and already working. We want to work smarter, not harder. There's no beneficial reason to reinvent the wheel here.

[koplo199]

It's not a matter of enabling options so much as it is disabling. Some options need to be disabled so that Valve's wine patched overrides can take effect.

That doesn't change anything to the question though. But in that case it seems weird that it is even necessary.

And yes, we could rebuild debian's packages with the same build flags then provide a ppa for them but it's more work for something that is already done (built in proton), and already working. We want to work smarter, not harder. There's no beneficial reason to reinvent the wheel here.

I already listed a fair amount of reasons of why this is a bad idea here and the beneficial reasons an alternative would bring here. If there's still something unclear I'll gladly add some clarification.

I'll add that there is no need for a ppa (even if it indeed would be the most elegant solution), and that rebuilding is only necessary in the case where Debian 11 is not used as the runtime base. Using Debian 11 would eliminate the "need" for a rebuild (even if it would be a lot cleaner to still do it).

To be clear: I'm willing to work on it, and I do not criticize the current solution for the sake of criticizing it. I know that in the real world we sometime have to go for a solution that is not optimal but works. This situation however does not fit under this category: it is broken by design, and works in the same sense a roofless house works when it is not raining.

[strycore]

@koplo199 thanks for looking into this!

Would it be possible to separate the Gstreamer runtime and ship it separately? I understand the Gstreamer libs have a different origin than what we use for the normal runtime but they also have a big chance of remaining the same between wine builds.
We don't have the ability to simply remove Gstreamer from current builds because that would break all existing installs.
But we can consider shipping Gstreamer (and other libs bundled with Wine) individually.

[strycore]

Going to merge this so I can start incorporating the Debian 12 runtime in the client. This is mostly going to be useful for native games and our other non wine runners.

[GloriousEggroll]

@koplo199 thanks for looking into this!

Would it be possible to separate the Gstreamer runtime and ship it separately? I understand the Gstreamer libs have a different origin than what we use for the normal runtime but they also have a big chance of remaining the same between wine builds. We don't have the ability to simply remove Gstreamer from current builds because that would break all existing installs. But we can consider shipping Gstreamer (and other libs bundled with Wine) individually.

the problem there is that users who turn off the lutris runtime then hit problems with gstreamer. right now by shipping the libraries with the build it forces the build to work regardless of whether or not they have the runtime enabled.

If we're going to go koplo's route with gstreamer then I propose we rebuild the gstreamer packages we need and provide a ppa for the buildbot to pull them from -and- include the libraries from those packages in our builds. The build options needed can be pulled from the proton-ge makefile for each gstreamer package:

https://github.com/GloriousEggroll/proton-ge-custom/blob/master/Makefile.in

Doing it this way will give us the same build options used in proton + debian 12 build/runtime base

[strycore]

I thought that disabling the runtime was also disabling bundled Gstreamer libs. Does having gstreamer in wine's lib folder forces them to load?

Disabling the runtime is not something that users should normally have to do. I want to replace the whole "disable runtime / prefer system libraries" with something a bit more controlled. Something like that:

[koplo199]

If we're going to go koplo's route with gstreamer then I propose we rebuild the gstreamer packages we need and provide a ppa for the buildbot to pull them from -and- include the libraries from those packages in our builds. The build options needed can be pulled from the proton-ge makefile for each gstreamer package:

https://github.com/GloriousEggroll/proton-ge-custom/blob/master/Makefile.in

Doing it this way will give us the same build options used in proton + debian 12 build/runtime base

I fully support the proposal. It would be both the solution offering the most freedom (by fully controlling to build configuration), and the cleanest one (by ensuring to build environment 100% match the runtime environment).

I thought that disabling the runtime was also disabling bundled Gstreamer libs. Does having gstreamer in wine's lib folder forces them to load?

As far as I know it doesn't: it won't be able to discover libraries not in LD_LIBRARY_PATH. I also confirm disabling the Lutris runtime also disable the bundled gstreamer libs: runtime not set -> never called -> never called.

Now that this PR has been merged, should we continue the discussion in a different place? (maybe by just opening an adequate issue?)

[GloriousEggroll]

A separate issue should be opened for the gstreamer stuff.

Also -- we decided to downgrade to Debian 11 after some users had issues because Debian 12 ships glibc 2.34, and ubuntu 20.04 ships 2.31, which is supported by canonical until 2030. The move to Debian 11 is also more in line with valve's proton runtime.

This means that this MR needs to be downgraded to Debian 11

[strycore]

While Canonical provides support until 2030, and only on servers probably, we usually don't support more than 2 LTS releases so we'll likely drop support for 20.04 when 24.04 comes out.

[GloriousEggroll]

While Canonical provides support until 2030, and only on servers probably, we usually don't support more than 2 LTS releases so we'll likely drop support for 20.04 when 24.04 comes out.

Debian is independant of Canonical -- not sure what you're referring to regarding this. 20.04 is coincidentally supported via it's glibc version as it's newer than the version Debian 11 ships, but overall at this point it's only coincidental.

[GloriousEggroll]

What I'm saying is that since Debian 11 ships glibc 2.31 -any- distro shipping glibc 2.31 or newer should work by sheer coincidence. The same will apply when we move to Debian 12 eventually with glibc 2.34. I've already verified this helps other people on ubuntu 20.04 (2.31), slackware 15 (2.33), and opensuse leap (2.31)

[strycore]

Our support window isn't based on anything other than a time frame, so we can adopt newer technologies while removing support from only a minimal number of users. This is done mostly to gain access to new Python or Gtk features for the client. The runtime is separate but it wouldn't make sense to provide support for older OSes when the client itself won't be supporting those platforms. I might send the glibc version in the initial API call that returns the list of runtimes so we can have some idea of the usage of given glibc versions but that won't be available until 0.5.15 or 0.6.0.

Right now, using Debian 11 as a base should be good enough, we can reconsider an update in May 2024.