Steam beta 2021-09-07: steamwebhelper cannot find LIBTHAI_0.1.25 on NixOS

[benneti]

Your system information

• Steam Runtime Version:
• Distribution (e.g. Ubuntu 18.04): NixOS 21.05
• Link to your full system information (Help -> System Information) in a Gist: https://gist.github.com/benneti/49b1e572b6af12d98805961723957a91
• (Additionally this happens on an Intel Laptop/AMD Desktop system (same OS )
• Have you checked for system updates?: Yes
• What compatibility tool are you using?: Steam Linux Runtime / Proton 5.13+
• If you are using Steam Linux Runtime for native Linux games: What versions are listed in SteamLinuxRuntime/VERSIONS.txt?
• If you are using Steam Linux Runtime, or Proton 5.13 or newer: What versions are listed in SteamLinuxRuntime_soldier/VERSIONS.txt?

#Name	Version		Runtime	Runtime_Version	Comment
depot	0.20210818.24			# Overall version number
pressure-vessel	0.20210809.1			
scripts	v0.20210818.0-0-gc6be089			# Entry point scripts, etc.
soldier	0.20210817.0	soldier	0.20210817.0	# soldier_platform_0.20210817.0/

Please describe your issue in as much detail as possible:

The issue described here ValveSoftware/steam-for-linux#7935 (comment) i.e. LIBRARY + SHOP only work in big picture mode because steamwebhelper is broken.

steamwebhelper.log

Steam runtime environment up-to-date!
Warning: An unexpected error occurred while executing "/sbin/ldconfig -XNv", the exit status was 127
./steamwebhelper: /home/benneti/.local/share/Steam/ubuntu12_64/steam-runtime-heavy/usr/lib/x86_64-linux-gnu/libthai.so.0: version `LIBTHAI_0.1.25' not found (required by /lib64/libpango-1.0.so.0)

Either the NixOS package needs to be changed to accomodate the missing libraries or they should be in the runtime.

Steps for reproducing this issue:

1. Install NixOS
2. Install Steam
3. Run Steam
4. Login and activate beta
5. Observe

[smcv]

@kisak-valve or @benneti, please could you retitle this to "Steam beta 2021-09-07: steamwebhelper cannot find LIBTHAI_0.1.25 on NixOS"? If people on different non-FHS distros like Guix have similar issues, then we'll want to track those separately.

[smcv]

@benneti, do you know who maintains the Steam package in NixOS? Can you ask them to participate in solving this issue?

[benneti]

I hope it is fine for you to be pinged here, could you maybe help @jagajaga @jonringer or maybe the maintainers of the runtime are more suited @hrdinka @abbradar ?

[smcv]

Steam for Linux is primarily designed to be run on a FHS-ish operating system like Ubuntu, Arch or Fedora, so it's always going to be a bit of an awkward fit on NixOS.

I know NixOS is not FHS, but I also know (from previous issues) that it builds a small FHS container and runs Steam inside that container, to make an environment a bit closer to Steam's expectations.

Some information that would help:

head -v ~/.steam/root/ubuntu12_64/steam-runtime-heavy.version.txt
head -v ~/.steam/root/ubuntu12_64/steam-runtime-heavy/version.txt
ls -l ~/.steam/root/ubuntu12_64/steam-runtime-heavy/pinned_libs_32/
ls -l ~/.steam/root/ubuntu12_64/steam-runtime-heavy/pinned_libs_64/

It might also be helpful to edit ~/.steam/root/ubuntu12_64/steam-runtime-heavy/run.sh and ~/.steam/root/ubuntu12_64/steam-runtime-heavy/setup.sh, and near the top of each one, where it says set -u, also add a line set -x; then try restarting Steam. That will put a lot more in steamwebhelper.log.

Warning: An unexpected error occurred while executing "/sbin/ldconfig -XNv", the exit status was 127

I think this is probably the root cause, or a large part of the root cause. On a more normal, FHS distribution, ldconfig would tell us all the directories that make up the default search path for the runtime linker, so that we could arrange the library search path correctly. What we want to end up with in the LD_LIBRARY_PATH is, in this order:

• the pinned_libs_32 and pinned_libs_64 subdirectories of steam-runtime-heavy
• your OS's default search path (and I know that isn't really a thing that exists on NixOS, but read on...)
• steam-runtime-heavy/usr/lib/x86_64-linux-gnu and steam-runtime-heavy/lib/x86_64-linux-gnu

However, there is clearly something a bit weird going on here, because we're successfully picking up libpango-1.0.so.0 from NixOS, but for some reason using our own copy of libthai.so.0 (which is too old for NixOS' pango). I would have expected that even in the case where ldconfig completely fails, we would either get a purely NixOS environment (which should work, as long as NixOS' FHS-ish container has all the libraries we need), or a purely heavy environment (which should also work, because since Steam beta 2021-09-xx we ship a self-contained heavy environment instead of the previous more ad-hoc arrangement), but not this strange mixture.

I think the easiest way to make this work might be something like this:

• in the FHS-ish container that NixOS uses to run Steam, make one directory (perhaps /usr/lib64, /usr/lib or /usr/lib/x86_64-linux-gnu) that has symlinks to all the x86_64 libraries that are provided;
• somehow tell the Steam Runtime launching scripts to treat that as your "default search path" instead of asking ldconfig;
• and change the launching scripts to pick that up.

[benneti]

I think it is a problem with the LD library path order (putting system library at the very end omitts this error, but then it mesa doesn't work as expected anymore).
Maybe the simplest solution is to remove the now obsolete libraries that weren't in the runtime previously, is there a list somewhere of the minimal set of libraries that should be in the nixos container? (i.e. stripping down https://github.com/NixOS/nixpkgs/blob/nixos-21.05/pkgs/games/steam/fhsenv.nix)

Additionally for completeness:

steam-run head -v ~/.steam/root/ubuntu12_64/steam-runtime-heavy.version.txt                                                                                     15:51
steam-run head -v ~/.steam/root/ubuntu12_64/steam-runtime-heavy/version.txt
steam-run ls -l ~/.steam/root/ubuntu12_64/steam-runtime-heavy/pinned_libs_32/
steam-run ls -l ~/.steam/root/ubuntu12_64/steam-runtime-heavy/pinned_libs_64/
==> /home/benneti/.steam/root/ubuntu12_64/steam-runtime-heavy.version.txt <==
steam-runtime-heavy_0.20210826.0
==> /home/benneti/.steam/root/ubuntu12_64/steam-runtime-heavy/version.txt <==
steam-runtime-heavy_0.20210826.0
total 0
total 0

(steam-run runs the following application in the environment steam sees)

[smcv]

I think it is a problem with the LD library path order (putting system library at the very end omitts this error, but then it mesa doesn't work as expected anymore).

The intended library path order is as I said: "pinned" libraries (those that are strictly newer in the runtime than on the host, or have had incompatible changes), then your libraries from NixOS' FHS container (which you can update), and finally our libraries from the heavy runtime (which we basically can't, unless we are willing to break older OSs, which we are not).

In your case there are no "pinned" libraries, which is expected when running on a reasonably up-to-date host OS.

The problem is that on NixOS, the "your libraries" part of the search order is only finding some of your libraries, but not the rest, for whatever reason.

If you add pango's dependencies (in particular libthai) to the FHS environment, so that they will appear in /lib64 rather than being loaded directly from /nix/store, does that work?

[benneti]

I already tried to add them, but then I simply hit the next library with such a problem.
Therefore I think it would be the simplest solution to remove all libraries from the environment that are already in the steam-runtime.

Thank you for the details by the way!

[smcv]

Therefore I think it would be the simplest solution to remove all libraries from the environment that are already in the steam-runtime.

Something else to consider here is: what would happen on a more typical OS that we actually have the resources to test, like Ubuntu or Arch? And if the answer on NixOS is "something totally different", then it's always going to be more likely to break on NixOS.

On a more typical OS, we're not running in a special minimal container, we're running on the host OS - so we can see all libraries, including those that the desktop environment depends on. The Steam Runtime is designed to work with this, rather than against it.

You've chosen to use an unusual non-FHS operating system, so in a sense you've brought this on yourself; but you can mitigate it by reducing the number of things visible to Steam that are unusual.

I already tried to add them, but then I simply hit the next library with such a problem.

Which is ...?

You can make this process take fewer iterations by assuming you are likely to need "most" of the libraries in heavy. There are actually not that many libraries in heavy - it's a small runtime specifically for the steamwebhelper - and probably you already have at least half of them. You can see them all by listing the contents of steam-runtime/heavy/{,usr/}lib/x86_64-linux-gnu.

[jonringer]

I made a PR to append the pinned libraries to LD_LIBRARY_PATH: https://github.com/NixOS/nixpkgs/pull/120829/files I could rebase the PR to see if it would fix anything, but:

the pinned_libs_32 and pinned_libs_64 subdirectories of steam-runtime-heavy
your OS's default search path (and I know that isn't really a thing that exists on NixOS, but read on...)
steam-runtime-heavy/usr/lib/x86_64-linux-gnu and steam-runtime-heavy/lib/x86_64-linux-gnu

just for clarity, this would be:

~/.steam/bin64/steam-runtime-heavy
~/.steam/bin32/steam-runtime-heavy
# default paths, these get mounted by nixos runtime container
/lib64
/lib32

I can't seem to find a steam-runtime-heavy/usr/lib/x86_64-linux-gnu anywhere.

[smcv]

This issue is about the Steam beta, not the general-availability version (at the moment). The general-availability version has a much simpler setup for steam-runtime-heavy that uses our bundled libraries unconditionally, but that causes compatibility issues for people with a sufficiently new version of Freetype installed (ValveSoftware/steam-for-linux#7935). The beta replaces it with something a lot more similar to the scout steam-runtime, which tries to use your libraries preferentially, and only falls back to our bundled libraries if it has to.

The intention is that for the steamwebhelper, after going through setup.sh and run.sh, we end up with the search path order I described. (The call path actually starts in ~/.steam/root/ubuntu12_64/steamwebhelper.sh, you can trace though shell scripts from there.)

To achieve that, I think all you would need to put in the caller environment should be LD_LIBRARY_PATH=/lib64:/lib32? Steam's scripts should be able to do the rest.

However, the way this currently works, and with your FHS environment not having a /sbin/ldconfig, we need every library that is both in heavy, and also provided by NixOS, to appear in one of those directories. For example, in the error message from this issue report, it is not enough to put libpango-1.0.so.0 in /lib64 and rely on getting libthai.so.0 loaded from /nix/store/... via a RUNPATH: to get the correct precedence order, we'd need to see libthai.so.0 in /lib64.

[benneti]

thanks john for joining in, was about to write an answer but smcv summarizes it better than I could

[smcv]

~/.steam/bin64/steam-runtime-heavy
~/.steam/bin32/steam-runtime-heavy

It is not supposed to be necessary to add these directories to the search path before running Steam. Any parts of Steam that need to search these directories are meant to edit the LD_LIBRARY_PATH themselves.

Looking at NixOS/nixpkgs#120829, it is not supposed to be necessary to add those directories to the search path either.

[jonringer]

Is there a way to get rid of using LD_LIBRARY_PATH altogether? I think part of the issue is that LD_LIBRARY_PATH is destructively forcing a lot of unwanted behavior as well.

For a while we were building a ldconfig.cache for steam to consume. Would there be another way to communicate dependency preference?

[smcv]

For a while we were building a ldconfig.cache for steam to consume

Are you not still doing that?

If you can provide a ldconfig cache, together with a /sbin/ldconfig that reads it, in the FHS-like container, then that would be the thing that's closest to what happens on more mainstream distros.

[smcv]

This issue is about the Steam beta, not the general-availability version (at the moment)

After today's update, the general-availability version is also likely to be affected by this.

The shortest path to a working package is probably one of these:

• generate a ldconfig cache in your FHS container (it would be /etc/ld.so.cache in most FHS distributions, but could be something different if you need it to be)
• provide a /sbin/ldconfig in your FHS container that will read that ldconfig cache and output a list of libraries in at least /lib32 and /lib64
• make sure all the libraries seen in steam-runtime-heavy/ are present in your FHS container's /lib64, not just somewhere below /nix/store

or

• run Steam with LD_LIBRARY_PATH=/lib64:/lib32 so that run.sh will search those directories after the pinned_libs_* directories, but before the rest of its search path
• make sure all the libraries seen in steam-runtime-heavy/ are present in your FHS container's /lib64, not just somewhere below /nix/store

If you are maintaining a Steam package in a distro, particularly one that is unusually-set-up and likely to experience compatibility issues, then I would recommend testing the beta client so that behaviour changes will not come as an unwelcome surprise when they appear in the general-availability branch.

[smcv]

Is there a way to get rid of using LD_LIBRARY_PATH altogether?

Sorry, this is not going to happen any time soon. The scripts based around LD_LIBRARY_PATH are far from ideal, but they're the best answer we currently have for portability to all distributions.

The most likely way to stop using LD_LIBRARY_PATH in the long term is to start running parts of Steam in containers, but that would make container support a mandatory dependency (not just for modern Proton, but for Steam itself), and we're not ready to have that dependency at the moment.

[tgharib]

Steam for Linux is primarily designed to be run on a FHS-ish operating system like Ubuntu, Arch or Fedora, so it's always going to be a bit of an awkward fit on NixOS.

FWIW, if the Steam flatpak was officially supported, I will gladly drop the NixOS native steam package and use that. Saves you guys the maintenance burden of supporting dozens of distros too. Just my 2c.

[smcv]

FWIW, if the Steam flatpak was officially supported, I will gladly drop the NixOS native steam package and use that.

The Flatpak app is not officially supported by Valve, but neither is NixOS, so the Flatpak app is certainly an option...

I would suggest weighing up the advantages and disadvantages of the Flatpak app vs. the NixOS-native package for yourself. Both can be summarized as putting Steam in a more-FHS-like container so that it won't be broken by NixOS' non-FHS layout.

[jbalme]

Both can be summarized as putting Steam in a more-FHS-like container so that it won't be broken by NixOS' non-FHS layout.

While true technically, in terms of design, Flatpak is not designed to act like a traditional FHS environment. It is FHS-like, but it intentionally breaks some things in the name of isolation and sandboxing.

NixOS's FHSEnv, on the other hand, is designed to run traditional FHS apps, and is not designed to be a sandbox. Something not working that works in the least common denominator of FHS distros, is a bug in NixOS's FHSenv imo. Unfortunately, it is currently a VERY leaky abstraction and there are many such bugs. I think at some point in the near future fhsenv will need to be redesigned entirely for a few reasons.

[YellowOnion]

(steam.override {
      extraPkgs = pkgs: with pkgs; [ pango harfbuzz libthai ];
    })

This fixed my issue.

@smcv The irony of all this DLL hell, is that nix is designed to solve this, if Steam and all sub components were built inside a pure nix derivation, Valve would pick up on any missing dll's pretty quickly, my biggest gripe is that the updates are pushed with no warning of changes, and don't seem to be widely tested, and no way to revert back to older versions, nix provides these facilities. If I break my ArchLinux system I have to find a rescue disk and then fix it, if I screw up my NixOS all I have to do is select a different generation from my boot menu.

This could be easily fixed if Valve used a sandboxing system to test their code, and then explicit defining what OS libraries are required for steam to build, using containers on the client end I don't think necessary. And then perhaps having a notice board somewhere that OS maintainers can quickly check and update their packages (maybe even parse), before these updates come out.

Yesterday Valve shipped this broken "beta" to everyone without waiting an extra few days to fix it for OS maintainers, why can't they wait a few extra days?

On top of that, it would be nice if there was a guide to debugging native steam games, Half-Life Seg-faults on my system, the dump files are instantly deleted and uploaded to Valves server, I can't figure out how to get a debugger attached inside the layers of runtimes, and even when I do get a crash dump file gdb doesn't seem compatible with them.

[anund]

Picking up on @smcv comment on it being weird to use both the nix version of libpango-1.0.so.0 and the steam version of libthai I looked at the contents of the current FHS used by steam. libpango-1.0.so.0 is there https://github.com/NixOS/nixpkgs/blob/master/pkgs/games/steam/fhsenv.nix#L126. libthai and harfbuzz are not. I suspect while the steamwebhelper version requirements lined up with the host system no one could see the partial sharing of a set of dependencies between bundled steam libraries and os.

Perhaps there's an assumption that the FHS container has a valid but minimized ecosystem of dependencies when it only has a minimum set added through trial and error. I suspect the short term fix is collecting the valid dependency set as a group and adding it to the FHS container on the nixos side.

Notes on steamwebhelper.log

Adding libthai to the nixos FHS container fixes ./steamwebhelper: /home/anund/data/Steam/ubuntu12_64/steam-runtime-heavy/usr/lib/x86_64-linux-gnu/libthai.so.0: version LIBTHAI_0.1.25 not found (required by /lib64/libpango-1.0.so.0)
Adding harfbuzz fixes ./steamwebhelper: symbol lookup error: /lib64/libpango-1.0.so.0: undefined symbol: hb_buffer_set_cluster_level
(both can be accomplished as pointed out just above)

[benneti]

as this will most likely be fixed on the nixos side (and there are already workarounds) feel free to close it here

[smcv]

Perhaps there's an assumption that the FHS container has a valid but minimized ecosystem of dependencies when it only has a minimum set added through trial and error. I suspect the short term fix is collecting the valid dependency set as a group and adding it to the FHS container on the nixos side.

To most closely resemble what is expected from a typical FHS distribution, the libraries in /lib64 on NixOS' FHS container should be self-contained (a "dependency closure"): so if your libpango requires libthai (which it does), then we should be able to find libthai in /lib64 too, not just in /nix/store. The same is true for the recursive dependencies of everything else in /lib64.

If the FHS environment does that, then that will pre-emptively fix various possible future issues.

my biggest gripe is that the updates are pushed with no warning of changes

I do not have control over Valve's update schedule for Steam. There are other regressions triggered by fixing ValveSoftware/steam-for-linux#7935 that I had hoped would be fixed in a new beta before going to GA, such as #458, but fixes for those didn't make it either.

why can't they wait a few extra days?

It wasn't my decision, but I would guess: either because ValveSoftware/steam-for-linux#7935 was breaking distributions that are more widely used than NixOS, or because changes in the Windows version of Steam were being released and the Linux version had to keep up.

Again, I would recommend that people who are involved in maintaining distros' Steam packages should use the Steam beta, so that changes that happen in the general-availability branch of Steam do not come as an unwelcome surprise when it's too late to work around them by switching between branches.

[benneti]

fixed in nixos