steam: Proton 5.13 does not launch because of steam runtime

[nyanloutre]

Describe the bug
Multiple errors when using SteamLinuxRuntime_soldier/run-in-soldier

This prevent the latest Proton (5.13) from running

To Reproduce
The following commands are to run in a steam-run shell: steam-run /bin/bash

1. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier -- echo coucou
   Error: x86_64-linux-gnu-capsule-capture-libs: code 2: open "/etc/ld.so.cache": No such file or directory
2. ldconfig -C /etc/ld.so.cache
3. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier -- echo coucou
   Error: /run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
4. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier --filesystem=/nix -- echo coucou

It's finally printing

Expected behavior
Should run natively from within steam and without manual tweaks

Additional context
There is two problems to solve:

1. How to generate the /etc/ld.so.cache file
2. How to permanently add the --filesystem=/nix option to steam bwrap

Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.69, NixOS, 21.03.git.0da2a1a113e (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(root): `"nixos-21.03pre246543.24c9b05ac53"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - steam
# a list of nixos modules affected by the problem
module:

[nyanloutre]

Related issues:

• Steam Linux Runtime not working on my system (Exherbo) ValveSoftware/steam-runtime#230
• NixOS system misconfiguration: ldconfig -p has exit status 1 #35387

[jhlkek]

Probably related, if I run Steam from a terminal window and try to run a game with Proton 5.13 I get this message before it (the game) crashes:

ERROR: ld.so: object '/home/user/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
ERROR: ld.so: object '/home/user/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

[smcv]

@nyanloutre, to clarify, if you apply these two workarounds (both of which are entirely unsupported and are done at your own risk, but they provide useful information):

• generate /etc/ld.so.cache
• hack the _start-container-in-background script to add --filesystem=/nix next to the other --filesystem options

is that sufficient to make Proton 5.13 work? (I suspect the answer is no, but I hope I'm wrong.)

How to permanently add the --filesystem=/nix option to steam bwrap

If it's that simple, then we can do the equivalent of that in the pressure-vessel source code. Making pressure-vessel work on non-FHS host distributions is not really on the radar right now (we have higher-priority issues to deal with, and fixes that apply to FHS distributions like Ubuntu benefit a lot more people than fixes for individual non-FHS distributions); but making a one-line change to share an extra filesystem has such a good effort/result and risk/result ratio that we can probably sneak it in to a future release anyway.

I'm not familiar with NixOS, except knowing that it's non-FHS. What's in /nix? Does it contain executables, libraries and static data belonging to OS-level packages, like /usr on a FHS system? Or does it contain system-specific variable data, like /etc, /var and /run on a FHS system? Or does it contain user data, like /home and /srv on a FHS system? Or some mixture of those?

If /nix only contains things that are reasonably similar to a FHS system's /usr, then we can certainly share it in the same situations where we'd share /usr.

If there are symbolic links into /nix from locations like /usr and /lib, are they absolute (/lib/foo -> /nix/path/to/foo), or relative (/lib/foo -> ../nix/path/to/foo), or a mixture? (This affects where we have to put it in the container to make it work: /nix, or /run/host/nix, or both.)

How to generate the /etc/ld.so.cache file

Where does a NixOS system normally keep its ld.so cache, and where will NixOS' glibc look for it?

Handling this "nicely" is likely to need C source code changes in libcapsule and pressure-vessel. See ValveSoftware/steam-runtime#230 (comment) and https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/40 - the exact paths involved on Exherbo and NixOS are almost certainly different, but the general idea will be the same.

We do not have the resources to support non-FHS distros, but I'm hoping to get infrastructure in place in the C code so that distro developers can contribute a patch that adds their filesystems/ld.so.cache/etc. to a reasonably obvious list (probably with Exherbo as the initial example, just because someone has already told me how it works).

[nyanloutre]

Hello, and thanks for the help !

@nyanloutre, to clarify, if you apply these two workarounds (both of which are entirely unsupported and are done at your own risk, but they provide useful information):

* generate `/etc/ld.so.cache`

* hack the `_start-container-in-background` script to add `--filesystem=/nix` next to the other `--filesystem` options

is that sufficient to make Proton 5.13 work? (I suspect the answer is no, but I hope I'm wrong.)

It still doesn't launch any games, I have this error:

ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

I'm not familiar with NixOS, except knowing that it's non-FHS. What's in /nix? Does it contain executables, libraries and static data belonging to OS-level packages, like /usr on a FHS system? Or does it contain system-specific variable data, like /etc, /var and /run on a FHS system? Or does it contain user data, like /home and /srv on a FHS system? Or some mixture of those?

In Nixos every binary and libs are located in /nix, everything else is only simlinks.
To make Steam work, NixOS is using bwrap to give it an FHS environment. Unfortunatly it's not enough to make the steam runtime work.

If there are symbolic links into /nix from locations like /usr and /lib, are they absolute (/lib/foo -> /nix/path/to/foo), or relative (/lib/foo -> ../nix/path/to/foo), or a mixture? (This affects where we have to put it in the container to make it work: /nix, or /run/host/nix, or both.)

The links are always absolute and point to /nix/store/xxxxxxxxxxx-libfoo/...

Where does a NixOS system normally keep its ld.so cache, and where will NixOS' glibc look for it?

NixOS doesn't have an ld.so cache, but it may be possible to generate it in the wrapper script that already launch steam in a bwrap.

[nyanloutre]

I also managed to get this error by adding unset LD_PRELOAD in the _v2-entry-point script:

/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so

[smcv]

ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

This is noise, you can safely ignore it. The way Steam sets up LD_PRELOAD is a bit weird, and will always cause these warnings.

/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

This is the real problem. The container doesn't have the required ld.so, or ld.so.cache, or search paths, or something, to find the dependencies of your libstdc++.so.6.

(I'm not sure why pressure-vessel-adverb is loading C++ code - it should only depend on GLib and libelf, which are C.)

To make Steam work, NixOS is using bwrap to give it an FHS environment

That might not work particularly well with the container runtime, which also relies on bwrap.

What does that FHS environment look like? Is /usr populated with symlinks into /nix?

NixOS doesn't have an ld.so cache

That's unfortunate, because libcapsule relies on parsing the glibc ld.so cache to be able to find host system libraries. The alternative would be to hard-code a list of library directories and hope it matches the locations your glibc will look in.

it may be possible to generate it in the wrapper script that already launch steam in a bwrap

I think that would be a good idea.

/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure

You can safely ignore this. GLib automatically loads GIO modules, and finds the version on the host system, for which the copy of GLib provided with pressure-vessel (which is from Steam Runtime v1 for maximum portability) is too old; but it doesn't actually need any GIO modules, so it's harmless that they fail to load. We actually try to turn them off, but until recently the included copy of GLib was too old to have a good mechanism to do this. A future release will probably stop this happening, but it isn't a high priority.

adding unset LD_PRELOAD

This will break the Steam overlay, which used to be just the overlay used for achievement popups, the Shift+Escape screen, etc. but is gradually accumulating more important functionality (game controller remapping, streaming, probably more).

[L-as]

I made an FHS like so:

with import <nixpkgs> {};
buildEnv {
	name = "steam-env";
	paths = [ gcc-unwrapped.lib glib.out json-glib.out ];
}

and added the lib of this FHS to the rpath of pressure-vessel-adverb, which seems to have gotten it further, but adding all of the required libraries causes it to fail on libstdc++ again! I checked with strace, and it fails with this:

1226  openat(AT_FDCWD, "/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

It opens libstdc++ from the rpath several times, but once tries to open it from here, but this output has no libstdc++, so it obviously fails. I am honestly not sure why it only tries this path, and I did try adding the FHS to LD_LIBRARY_PATH too to no effect.

[L-as]

@nyanloutre I can't reproduce your results with run-in-soldier.
For me, the following command: steam-run sh -c ldconfig -C /etc/ld.so.cache -f /dev/null /lib; $HOME/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier --filesystem=/nix -- echo test gives the following output:

/nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure
Failed to load module: /nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so
pressure-vessel-wrap[4]: Using glibc from provider system for some but not all architectures! Arbitrarily using provider locales.
pressure-vessel-wrap[4]: Using libdrm.so.2 from provider system for some but not all architectures! Will take /usr/share/libdrm from provider.
/run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: /../lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.30' not found (required by /nix/store/8nn8q8p6frw05q6d87kdig8x3c4pxly6-libselinux-2.9/lib/libselinux.so.1)

Wrt. to my above halfway workaround, even adding an ld.so.cache doesn't make it load from libstdc++ from another place. It always tries to load it from the same folder where the interpreter is located.

[L-as]

@smcv wrt. the FHS environment, it's actually /lib, /bin, etc. that are filled with absolute symlinks to /nix/..., and then /usr/x is just a symlink to /x for any x.

[smcv]

/nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure

You can ignore that, it's harmless.

pressure-vessel-wrap[4]: Using glibc from provider system for some but not all architectures! Arbitrarily using provider locales.
pressure-vessel-wrap[4]: Using libdrm.so.2 from provider system for some but not all architectures! Will take /usr/share/libdrm from provider.

Do you have both 32- and 64-bit graphics drivers on the host system? To use Steam, you really should (I'm surprised it works without involving pressure-vessel). The container has some basic Mesa graphics drivers (more as an accident of the dependency stack than anything deliberate), but they're from Debian 10, so they won't support very recent hardware.

/run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: /../lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.30' not found (required by /nix/store/8nn8q8p6frw05q6d87kdig8x3c4pxly6-libselinux-2.9/lib/libselinux.so.1)

That's a new error, so we're making progress!

Please could you get a log with PRESSURE_VESSEL_VERBOSE=1 in the environment? That will make pressure-vessel give a lot more info about what it's thinking.

wrt. the FHS environment, it's actually /lib, /bin, etc. that are filled with absolute symlinks to /nix/..., and then /usr/x is just a symlink to /x for any x

That's the reverse of the /usr merge. pressure-vessel should cope equally well either way round, but if /usr was populated with a directory hierarchy (with symlinks to the libraries in /nix) and the compat symlinks were lib -> usr/lib, etc., that might get you onto more-commonly-tested code paths.

[moben]

For reference, here is a log with with PRESSURE_VESSEL_VERBOSE=1:

https://gist.github.com/moben/26dfe84ea4208bee42d98f40dd180c40

Full list of changes to get to this point:

1. Added /bin/ldconfig -C /etc/ld.so.cache to the top of SteamLinuxRuntime_soldier/_start-container-in-background.
2. Added --filesystem=/nix to the other --filesystem parameters in SteamLinuxRuntime_soldier/_start-container-in-background.
3. exported PRESSURE_VESSEL_VERBOSE=1 in SteamLinuxRuntime_soldier/_v2-entry-point as it wasn't propagated when set as a launch option in steam.

pressure-vessel-launcher[669]: Child 715 died: wait status 32512
pressure-vessel-launch[702]: child 715 exited: wait status 32512
pressure-vessel-launch[702]: child exit code 715: 127
pressure-vessel-launch[702]: Exiting with status 127
/data/src/clientdll/installscript_posix.cpp (419) : Assertion Failed: Standalone evaluator returned error code for app 418530
/data/src/clientdll/installscript_posix.cpp (419) : Assertion Failed: Standalone evaluator returned error code for app 418530

seems relevant. Looks like either libexpat is missing from the container that steam runs in on NixOS to simulate something close to an FHS environment or a python3 from the runtime is called but can't find the lib from the runtime?

Edit:
Tried adding libexpat to the environment the steam client runs in, but the error remains. (might have made a mistake when adding the lib, moben@eada9fa)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/steam-games-unable-to-load-libstdc-so-6/9589/6