Moving Steam Linux Runtime causes Proton games to instantly fail and not log any errors

[hasezoey]

Your system information

• Steam client version (build number or date): 1646527495
• Distribution (e.g. Ubuntu): Manjaro 21.2.4
• Opted into Steam client beta?: [Yes/No] yes
• Have you checked for system updates?: [Yes/No] yes

Please describe your issue in as much detail as possible:

Currently Steam Linux Runtime - Soldier when moved to a different steam library fails to start any Proton game (that has worked before moving)

also, it is not uninstallable to be able to re-install it (see error at the end)

Steps for reproducing this issue:

1. Install Tool to a steam library (in my case it was sshfs)
2. Open a Proton Game (a known working one)
3. Observe Game working
4. Close Game
5. Move Tool to different steam library (in my case it was local btrfs)
6. Try to open a Proton Game (same as before)
7. Observe the game not working
8. Move Tool back to old steam library (in my case the sshfs one)
9. Open a Proton Game (same as before)
10. Observe Game working again

Tool = Steam Linux Runtime - Soldier

---

Also i cannot uninstall the Tool and try to re-install it because of Error: missing shared Content

PS: i dont have any logs for this, because Proton does not even go so far as to start logging and if there are steam launcher logs, i have not found them, also i have not found any dumps

[smcv]

Move Tool to different steam library

Was this via Properties -> Local Files -> Move install folder..., or by moving files manually?

I was unable to reproduce this problem by using Properties -> Local Files -> Move install folder... with beta 1646806235.

If you move the files manually, then it seems reasonable that Steam will not know where to find the new location of the compatibility tool (how would it know?) and therefore launching will fail.

if there are steam launcher logs, i have not found them

If you run Steam from a terminal (gnome-terminal, xterm, konsole or equivalent), every time you launch a game it will log what command you are trying to launch. For a Proton game, the command includes the soldier container runtime wrapper, the Proton wrapper, and the game itself.

Proton does not even go so far as to start logging

If Steam can't find the compatibility tool, then Proton will never start, so it can't possibly produce logs.

[hasezoey]

Was this via Properties -> Local Files -> Move install folder..., or by moving files manually?

sorry if this was not clear, i definitely meant via steam (Move install folder)

If you run Steam from a terminal (gnome-terminal, xterm, konsole or equivalent)

are you sure? because every time i have launched steam via terminal it either forked itself or just printed ~3 lines at startup and then never again, do i need a special argument like --debug?

(for reference i had tried that on manjaro, debian, ubuntu, linux mint), though some of them were at different (older) times

[smcv]

every time i have launched steam via terminal it either forked itself or just printed ~3 lines at startup and then never again

If Steam is already running, then the second instance will just send a message to the first instance, then exit. You'll need to completely exit from Steam, so that the instance you launch from a terminal is the first one you ran.

[smcv]

What you should see when you launch a game is something like this:

/bin/sh\0-c\0/home/me/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=312990 --
'/home/me/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun --
'/home/me/SteamLibrary/steamapps/common/Proton - Experimental'/proton waitforexitandrun
'/home/me/SteamLibrary/steamapps/common/Broforce The Expendables Missions/Expendabros.exe'\0

except it'll all be one line.

If Steam has lost track of where the compatibility tool is installed, then you'll probably get a "No such file or directory" error message shortly afterwards.

I tried moving Steam Linux Runtime - soldier from my custom library ~/SteamLibrary to the default library in Steam's installation directory, and that worked OK: Steam used the new path to SteamLinuxRuntime_soldier to launch the game.

[hasezoey]

If Steam is already running, then the second instance will just send a message to the first instance, then exit. You'll need to completely exit from Steam, so that the instance you launch from a terminal is the first one you ran.

i know, i meant it when no other instance was started

so as clarification: that was the experience i had last time i tried, and i now tried again - and my console was full of logs (i think the last time i tried debugging steam with logs was mid 2021)

---

I have now run steam with logging, moved the tool, tried a (few) games, and then moved it back and exited, here are the lines i think may be errors / warnings:

ERROR: ld.so: object '/home/hasezoey/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.

pressure-vessel-wrap[218815]: E: Could not create copy "./bin/[" from "/mnt/projects/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/soldier_platform_0.20220131.0/files/./bin/[" into "/tmp/tmp-L3VOI1/usr": Setting xattrs: fsetxattr(btrfs.compression): Operation not supported

Full Steam Log

[smcv]

ERROR: ld.so: object '/home/hasezoey/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.

You can safely ignore this, it's an oddity of how Steam gets the overlay loaded.

[smcv]

pressure-vessel-wrap[218815]: E: Could not create copy "./bin/[" from "/mnt/projects/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/soldier_platform_0.20220131.0/files/./bin/[" into "/tmp/tmp-L3VOI1/usr": Setting xattrs: fsetxattr(btrfs.compression): Operation not supported

This is a bug in the container runtime, but I don't quite understand what's happening there: there are at least two things wrong.

The first problem is that the container setup tool (pressure-vessel) is copying the container runtime into /tmp/tmp-(something). It isn't normally meant to do that: it should be creating a temporary copy in SteamLinuxRuntime_soldier/var/tmp-(something) instead (which will be much, much faster than using /tmp, because it uses hard-links or btrfs reflinks to avoid actually copying any data). So the first thing to figure out is why that's happening.

The second problem is that when copying from SteamLinuxRuntime_soldier into /tmp, the copy is failing because it's trying to copy an unsupported extended attribute. It shouldn't be trying to copy extended attributes here (there's no point).

[smcv]

Have you modified the SteamLinuxRuntime_soldier directory or its configuration? For instance, did you make SteamLinuxRuntime_soldier/var a symlink to /tmp, or edit any of the files in the SteamLinuxRuntime_soldier directory, or change any PRESSURE_VESSEL_* environment variables such as PRESSURE_VESSEL_VARIABLE_DIR or PRESSURE_VESSEL_COPY_RUNTIME_INTO?

If you have made any modifications like those, please undo those modifications before continuing.

---

If undoing any local modifications doesn't fix this, with SteamLinuxRuntime_soldier still on btrfs, please exit from Steam completely, and re-run it as something like this:

STEAM_LINUX_RUNTIME_VERBOSE=1 steam 2>&1 | tee steam.log

Then try launching a Proton game. This will produce a lot more output in the terminal and in steam.log.

(Normally I'd be asking you to use STEAM_LINUX_RUNTIME_LOG=1 - but the log file is written to the same variable directory as the temporary runtime, so if the variable directory is getting chosen in some weird way, I won't be able to predict where the log would end up!)

[hasezoey]

Have you modified the SteamLinuxRuntime_soldier directory or its configuration? For instance, did you make SteamLinuxRuntime_soldier/var a symlink to /tmp

no i have not messed with this tool's install at all (like i said in the original i would actually like to completely reset it, which is not possible, because i cannot uninstall because of Error: missing shared Content)

or change any PRESSURE_VESSEL_* environment variables such as PRESSURE_VESSEL_VARIABLE_DIR or PRESSURE_VESSEL_COPY_RUNTIME_INTO?

there are none set that i manually set

Then try launching a Proton game.

while the tool is moved to where it currently fails or where it is working?

---

also, i dont know if it means something but /tmp is a tmpfs on my system

also, when it works, the tool is creating a directory/tmp/tmp-xxxx which contains a basic linux system with symlinks to somewhere else (currently not on the machine, so i cannot provide a output of it today anymore)

[smcv]

while the tool is moved to where it currently fails or where it is working?

Whichever is easier, the information I'm looking for is mostly going to be before it fails.

The output of ls -l /mnt/projects/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier (or wherever you have moved it to, if not there) would also be very useful to see.

when it works, the tool is creating a directory/tmp/tmp-xxxx which contains a basic linux system with symlinks to somewhere else

That sounds like pressure-vessel has somehow decided to use /tmp instead of SteamLinuxRuntime_soldier/var as its temporary directory for variable data. I'm surprised it's doing this, and I want to find out why, because this will be making the container setup much slower for you than it should be.

The "basic Linux system" you mention is a copy of the soldier container runtime, altered to use your host system's graphics drivers. It's intentional that this is created, but it's meant to be in SteamLinuxRuntime_soldier/var/tmp-xxxx, not in /tmp - and because SteamLinuxRuntime_soldier/var is on the same filesystem as SteamLinuxRuntime_soldier/soldier_*, the "copying" is meant to be done with hardlinks or btrfs reflinks, which means that when it's working correctly, it doesn't actually take up any extra space.

Also i cannot uninstall the Tool and try to re-install it because of Error: missing shared Content

The container runtime doesn't control how Steam installs and uninstalls it, so that would be a separate issue (in ValveSoftware/steam-for-linux rather than ValveSoftware/steam-runtime). Please gather enough information for me to diagnose this one before you do anything about that, because I want to understand what has gone wrong while the broken installation is available for inspection.

[hasezoey]

ls -al original:

total 48
drwxr-xr-x 1 hasezoey hasezoey  240 Mar  9 20:12 .
drwxrwxr-x 1 hasezoey hasezoey  436 Mar  9 20:10 ..
drwxr-xr-x 1 hasezoey hasezoey   42 Mar  9 20:10 pressure-vessel
-rwxr-xr-x 1 hasezoey hasezoey 2480 Aug 25  2021 README.md
-rw-r--r-- 1 hasezoey hasezoey    0 Aug 26  2021 .ref
-rwxr-xr-x 1 hasezoey hasezoey  590 Feb  9 20:41 run
-rwxr-xr-x 1 hasezoey hasezoey  590 Feb  9 20:41 run-in-soldier
drwxr-xr-x 1 hasezoey hasezoey   58 Mar  9 20:12 soldier_platform_0.20220131.0
-rwxr-xr-x 1 hasezoey hasezoey  162 Aug 25  2021 toolmanifest.vdf
-rwxr-xr-x 1 hasezoey hasezoey 7394 Jan  4 18:30 _v2-entry-point
lrwxrwxrwx 1 hasezoey hasezoey    5 Mar  9 20:12 var -> /tmp/
-rwxr-xr-x 1 hasezoey hasezoey  264 Feb  9 20:41 VERSIONS.txt

ls -al moved:

total 32
drwxr-xr-x 1 hasezoey hasezoey  240 Mar 10 11:50 .
drwxr-xr-x 1 hasezoey hasezoey  352 Mar 10 11:49 ..
drwxr-xr-x 1 hasezoey hasezoey   42 Mar 10 11:49 pressure-vessel
-rwxr-xr-x 1 hasezoey hasezoey 2480 Aug 25  2021 README.md
-rw-r--r-- 1 hasezoey hasezoey    0 Aug 26  2021 .ref
-rwxr-xr-x 1 hasezoey hasezoey  590 Feb  9 20:41 run
-rwxr-xr-x 1 hasezoey hasezoey  590 Feb  9 20:41 run-in-soldier
drwxr-xr-x 1 hasezoey hasezoey   58 Mar 10 11:49 soldier_platform_0.20220131.0
-rwxr-xr-x 1 hasezoey hasezoey  162 Aug 25  2021 toolmanifest.vdf
-rwxr-xr-x 1 hasezoey hasezoey 7394 Jan  4 18:30 _v2-entry-point
lrwxrwxrwx 1 hasezoey hasezoey    5 Mar 10 11:49 var -> /tmp/
-rwxr-xr-x 1 hasezoey hasezoey  264 Feb  9 20:41 VERSIONS.txt

i recall having at some point in time to have done a rm -rf var/ because it was filled with many tmp-xxxx directories that all consumed some space (remember, this was on a sshfs filesystem, where i dont know if hardlinks are supported or only emulated)
but i dont quite recall having done a symlink

---

the following logs were ran with STEAM_LINUX_RUNTIME_VERBOSE=1 steam 2>&1 | tee steam.log

Steam Log for original location
Steam Log after having moved

---

i had also many times already tried to validate all files with steam (built in Verify integrity of tool files) which ran without errors, and i also could not uninstall the tool, like mentioned earlier to get a fresh set of files

---

after having removed the var symlink, it starts without problems, though now var/ contains residual tmp-xxx directories again (which still consume some space, which can accumulate)

[smcv]

i recall having at some point in time to have done a rm -rf var/ because it was filled with many tmp-xxxx directories [...] but i dont quite recall having done a symlink

pressure-vessel doesn't create this symlink itself, so you must have created it at some point. Deleting var/ is fine (it will be re-created if necessary), but replacing it with a symlink onto a different filesystem will make the container framework slow and unreliable.

Replacing anything that is normally owned by you with a symlink to a world-writeable directory (/tmp, /var/tmp, /dev/shm) can also create security vulnerabilities, so please avoid doing that.

I'm tempted to make pressure-vessel detect var being a symlink and just delete the symlink, and people who really, really want to redirect var onto another filesystem (at their own risk!) can use environment variables or bind-mounts.

contains residual tmp-xxx directories again (which still consume some space, which can accumulate)

They are not meant to accumulate, unless your games do not exit. Every time a new pressure-vessel instance starts up, it cleans up any tmp-xxx directories that are no longer in use, so you should only ever have one directory like this. I'll check your log files and see whether something is breaking this, but it is meant to work.

When hard-links are working correctly, each tmp-xxx directory should be 99% directories, symlinks, and hard-links into soldier_platform_*/ or pressure-vessel/. Exceptions: empty files are created anew instead of being hard-linked together, and usr/lib/pressure-vessel/overrides/share/vulkan/ contains a few small JSON files (hundreds of bytes each).

du can show you this: even with a pessimistic estimate of 4K of overhead per unique file (modern filesystems do better than this), it still only thinks the temporary directory has 7.5M of files that didn't already exist in the downloaded depot:

$ du -sh var/tmp-*/
626M    var/tmp-2L7XI1/
$ du -sh pressure-vessel/ soldier_platform_*/ var/tmp-*/
2.6M    pressure-vessel/
625M    soldier_platform_0.20220131.0/
7.5M    var/tmp-2L7XI1/

We have to do this setup every time because you might have installed, uninstalled or upgraded system packages, which would invalidate the decisions we made last time about which libraries to take from the host system; and we have to use a different directory every time because we can't guarantee that two containers aren't being launched simultaneously.

remember, this was on a sshfs filesystem, where i dont know if hardlinks are supported or only emulated

Steam and Steam games are I/O-intensive, so I would strongly recommend keeping your Steam library on a fully-featured local filesystem (such as ext4, btrfs or xfs). Removing the var symlink is an appropriate way to resolve the issue you were seeing.

However, if the target directory supports hard links, then hard links within a mount point seem to work fine on sshfs. The inode number and link count don't correctly show up as hard-linked together in stat or ls on the client, but if you look at the inode number and link count on the server, the files are actually hard-linked together rather than being copied. (Tested with sshfs_3.7.1+repack-2 from Debian.)

$ sshfs test-vm:/home/user ~/mnt
$ ln ~/mnt/test ~/mnt/test2
$ ls -ali ~/mnt/test*
238 -rw-r--r-- 1 smcv smcv 6 Mar 10 12:37 /home/smcv/mnt/test
239 -rw-r--r-- 1 smcv smcv 6 Mar 10 12:37 /home/smcv/mnt/test2
$ ssh test-vm "ls -ali ~/test*"
3932616 -rw-r--r-- 2 user user 6 Mar 10 12:37 /home/user/test
3932616 -rw-r--r-- 2 user user 6 Mar 10 12:37 /home/user/test2

[smcv]

From your log, it looks like the var/tmp-* cleanup is being done:

pressure-vessel-wrap[41429]: D: Cleaning up old subdirectories in /mnt/projects/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/var...

and also

steam-runtime-tools-Message: Profiling: start: Cleaning up temporary runtimes in /tmp
...
pressure-vessel-wrap[41429]: D: Found temporary runtime /tmp/tmp-GL2KI1, considering whether to delete it...
pressure-vessel-wrap[41429]: D: Deleting "/tmp/tmp-GL2KI1"...

So there should only be one var/tmp-* directory at a time, unless you launch a game that does not fully exit (for example Ubisoft's Assassin's Creed series tends to leave the Uplay launcher running in the background). We deliberately don't delete var/tmp-* if they are still in use, as tracked by a pressure-vessel-adverb process holding a lock on a file var/tmp-*/.ref.

[hasezoey]

(for example Ubisoft's Assassin's Creed series tends to leave the Uplay launcher running in the background).

i know, i always quit uplay after a session (and even if not, it will be closed the next reboot happens)

From your log, it looks like the var/tmp-* cleanup is being done:

can confirm that it actually cleans things up, the last time i actually had checked this was mid 2021 i think, where it didnt clean it up (even after reboots with no games / tools running)

---

so i would say this is is "user error", but maybe some errors (without extra logging) would be helpful

[smcv]

so i would say this is is "user error", but maybe some errors (without extra logging) would be helpful

I agree. What I'm intending to do about this:

• if var is a symlink, log a warning and delete it
• if we can't hard-link all files into the temporary directory, log a warning like "cannot hard-link files, falling back to copying: this will be slow and take up more space"
• and if we fall back to copying, don't try to copy extended attributes

[smcv]

What I'm intending to do about this

... is in the queue to be reviewed for a future SLR update.

@kisak-valve, I think we can close this issue, since the reporter concluded that it was user error.

[smcv]

Also i cannot uninstall the Tool and try to re-install it because of Error: missing shared Content

@hasezoey, I don't need any more information from you for this (I was able to reproduce the same failure modes on a test system); so if you want to report the failure to uninstall/reinstall as a separate issue in ValveSoftware/steam-for-linux, you can do that now.

Inability to uninstall/reinstall a Steam depot would be a problem with the Steam content downloader, rather than a problem with the specific content that you are uninstalling/reinstalling, so it would be a steam-for-linux issue rather than a steam-runtime issue (I know you originally reported this one against steam-for-linux too, but it was moved). There might be more information in ~/.steam/root/logs/content_log.txt.

You shouldn't be able to uninstall soldier while you have Proton games installed (because that would make Proton unable to launch), but I would have expected that the error message should be more like «Cannot uninstall "Steam Linux Runtime - Soldier" because it is required by "Proton 7.0"».

[hasezoey]

thanks for the help, i have also opened up the issue for being unable to uninstall it as ValveSoftware/steam-for-linux#8473

[smcv]

• if var is a symlink, log a warning and delete it
• if we can't hard-link all files into the temporary directory, log a warning like "cannot hard-link files, falling back to copying: this will be slow and take up more space"
• and if we fall back to copying, don't try to copy extended attributes

The SteamLinuxRuntime_soldier and SteamLinuxRuntime_sniper betas now have these changes. VERSIONS.txt will say pressure-vessel 0.20220315.0 or later.