5.13 and Experimental don't see symlinks

[xDShot]

Games in the Proton can't read symlinked folders and files. WIne Explorer can't see them as regular files and folders as well.

[kisak-valve]

Hello @xDShot, starting with Proton 5.13, Proton is run inside the Steam Linux Runtime - Soldier container environment and the container environment does not have unlimited access to the host system. Please clarify exactly what folders are involved with the symlink. Let's treat this as a Pressure Vessel issue until there's a stronger indication the issue is elsewhere.

The discussion on #308 may be helpful.

[xDShot]

@kisak-valve
After some experiments, turned out symlinks don't work in Proton if they target to files not on same hard drive partition.

[kisak-valve]

Right, so as discussed in #308, you probably need to start Steam with something like STEAM_COMPAT_MOUNTS=/path/to/unexpected/folder steam for Proton to be able to see it inside the container.

[apocalyptech]

Aha, yeah, just ran into this myself, after I'd moved Borderlands 3 from Proton 5.0 to 5.13. I always dislike having savegames/configs stored under compatdata (since I've had things like game uninstalls totally wipe compatdata/<gameid> dirs in the past, and also I like being able to wipe them manually if I accidentally screw something up and want Steam to rebuild the steamroot from scratch), so I've had a symlink from /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3 to /games/bl3_steam/userdir.

With Pressure Vessel, access to the dir was failing, as I found in an strace:

[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3", {st_mode=S_IFLNK|0777, st_size=24, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved", 0x1c59e880) = -1 ENOENT (No such file or directory)

... but launching Steam with STEAM_COMPAT_MOUNTS=/games steam fixed that access right up. Thanks!

One difference in my situation -- compared to xDShot's mention of symlinks don't work in Proton if they target to files not on same hard drive partition -- is that all of the game install, steamroot, and symlink destination were all contained under the single partition mounted at /games, so I don't think that partitioning has anything to do with it really.

[apocalyptech]

Huh, well, okay, I guess STEAM_COMPAT_MOUNTS=/games steam didn't entirely fix things up for me. It allowed the game to read its profile/savegame data and get into the game, but it seems unable to actually save data to that save dir. For instance, in that BL3 savegame dir, I've now got these files:

-rw-rw-r-- 1 bl3 bl3  698085 Jan  4 00:27 10.sav
-rw-rw-r-- 1 bl3 bl3  149117 Jan  4 00:27 profile.sav
-rw-rw-r-- 1 bl3 bl3  698084 Jan 14 21:52 10.tmp
-rw-rw-r-- 1 bl3 bl3  149117 Jan 14 21:52 profile.tmp

The top two on Jan 4 were from the last time I'd started this up (on an old version of Proton), and these new .tmp files are the game's attempt to write out new save/profile data for the game. Clearly the call to move those things on top of the old files are failing, though. I also see that 397540/pfx/dosdevices/c: has a bunch of temp files being created in it, like:

-rw-rw-r--  1 bl3 bl3    0 Jan 14 21:51  rf6b05.tmp
-rw-rw-r--  1 bl3 bl3    0 Jan 14 21:51  rf6ef5.tmp
-rw-rw-r--  1 bl3 bl3    0 Jan 14 21:51  rf72e2.tmp

Doing a bit more strace action I can see that the game's looking for these files but apparently not finding them:

[pid 834483] 21:51:42 lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/rf6ef5.tmp", 0x1c7ce2a0) = -1 ENOENT (No such file or directory)

So clearly something's still not quite right...

[smcv]

Symbolic links and crossing container boundaries do not go together particularly well, because entering the container changes the interpretation of the symlink: /games inside the container is not the same place as /games in the host system, unless we take steps to make it the same, which we don't necessarily know we need to do. This is what STEAM_COMPAT_MOUNTS is for - it tells pressure-vessel "put these directories in the container, even if you think there is no reason to".

Combine this with Proton, which is trying to pretend that there is no such thing as a symlink for better compatibility with Windows programs, and what you get is mostly a mess.

pressure-vessel is also designed to be able to limit the amount of stuff on the host system that gets shared with the container, as part of an experimental feature to give each game its own mock home directory (similar to Flatpak apps). That feature is not actively in use, but the fact that it is in our goals for the future means we're reluctant to be too indiscriminate about mirroring things from the host system in the container, because after we have made a directory be shared, changing it to not be shared carries a risk of regression (compatibility break).

If you think pressure-vessel is doing the wrong thing, we will need a log that says what it's doing and what it's thinking: see https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md for details.

Alternatively, if you are offloading directories onto a different drive, bind-mounts are often more robust than symlinks. When pressure-vessel makes a directory available in the container, it will do the same for any mount points below that directory, recursively.

For your use-case of preserving saved games in the face of a malfunctioning or uninstalled game or a need to wipe the compatdata directory, another option is that you might be better off with a backup/restore strategy: back up the saved games periodically to some unrelated location that Proton/Wine can't see (perhaps even a location that pressure-vessel can't see!), and know that they are not going to get deleted from there by the game or by Steam.

I don't think that partitioning has anything to do with it really

You are correct. pressure-vessel works at a level of individual directories, rather than mount points on the host system. However, when @xDShot said "if they target to files not on same hard drive partition", that is the most common scenario in which you will find that a symlink is made available in the container but its target directory is not.

[apocalyptech]

/games inside the container is not the same place as /games in the host system, unless we take steps to make it the same, which we don't necessarily know we need to do. This is what STEAM_COMPAT_MOUNTS is for - it tells pressure-vessel "put these directories in the container, even if you think there is no reason to"...

If you think pressure-vessel is doing the wrong thing, we will need a log that says what it's doing and what it's thinking: see https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md for details.

Okay, I'll go through there in a bit and see if I can get some detailed logs about it; I suspect that something's a bit funky with how it's handling these things. Though as you say, maybe it's the symlink itself which is causing fits...

Alternatively, if you are offloading directories onto a different drive, bind-mounts are often more robust than symlinks. When pressure-vessel makes a directory available in the container, it will do the same for any mount points below that directory, recursively.

Yeah, I was doing some bind-mounting earlier, 'cause my /games mount actually used to be /usr/local/games, which of course pressure-vessel didn't like, either (I know that's not exactly kosher, but in my defense, I've had that partition since 2002). I'd considered it for this case as well, but bind mounts really make me itchy. I hate having clutter in my /etc/fstab, having extra junk in my mount/df output, etc... I've got a long history of being able to use symlinks to get data shuffled about how I want, so it's been a bit of a journey to figure out how this fits into having suddenly-container'd apps running on here. :) For now I'd be happiest getting STEAM_COMPAT_MOUNTS working, but in the absence of that I might fail back to bind mounts anyway.

For your use-case of preserving saved games in the face of a malfunctioning or uninstalled game or a need to wipe the compatdata directory, another option is that you might be better off with a backup/restore strategy

Heh, yeah, I do have my nightly backups which pick these up, and provide me with a month of incrementals. I generally try to make sure I'm in a position where I never actually need the backups, though. :) On a semi-related note, I'd been meaning to submit something to suggest a feature request that maybe the wineroot's My Documents should default to some symlink inside the user's homedir. I know that some Codeweavers products back in the day did that, and it was kind of handy having that stuff default to being outside the wineprefix. Looks like I've still got dirs for Fallout3, Oblivion, and Sid Meier's Pirates! in my ~/Documents/My Games dir.

[apocalyptech]

Okay, so here's the full "System Information" output: system_info.txt

... and here's the debug logs while running the game, having been launched with STEAM_COMPAT_MOUNTS=/games STEAM_LINUX_RUNTIME_LOG=1 PROTON_LOG=1 PRESSURE_VESSEL_VERBOSE=1: slr-app397540-s940fed468257c98a.log

Let me know if I can provide any more info! Thanks again.

[smcv]

So, we now have, among other things:

10:32:06.213153: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213172: pressure-vessel-wrap[1131709]: 	'/games'
10:32:06.213190: pressure-vessel-wrap[1131709]: 	'/games'
10:32:06.213209: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213228: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps'
10:32:06.213247: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps'
10:32:06.213266: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213285: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/Borderlands 3'
10:32:06.213304: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/Borderlands 3'
10:32:06.213323: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213342: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/Proton 5.13'
10:32:06.213361: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/Proton 5.13'
10:32:06.213389: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213409: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/SteamLinuxRuntime_soldier'
10:32:06.213429: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/SteamLinuxRuntime_soldier'
10:32:06.213452: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213475: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel'
10:32:06.213500: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel'
10:32:06.213524: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213549: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/compatdata/397540'
10:32:06.213573: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/compatdata/397540'
10:32:06.213596: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213619: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/shadercache/397540'
10:32:06.213643: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/games/steamapps/shadercache/397540'
10:32:06.213665: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213688: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root'
10:32:06.213711: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root'
10:32:06.213733: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213756: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/linux32'
10:32:06.213778: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/linux32'
10:32:06.213801: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213823: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/linux64'
10:32:06.213846: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/linux64'
10:32:06.213868: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.213892: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32'
10:32:06.213915: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32'
10:32:06.213937: pressure-vessel-wrap[1131709]: 	'--ro-bind'
10:32:06.213962: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32/libVkLayer_steam_fossilize.so'
10:32:06.213986: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32/libVkLayer_steam_fossilize.so'
10:32:06.214009: pressure-vessel-wrap[1131709]: 	'--ro-bind'
10:32:06.214032: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32/steamoverlayvulkanlayer.so'
10:32:06.214056: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_32/steamoverlayvulkanlayer.so'
10:32:06.214080: pressure-vessel-wrap[1131709]: 	'--bind'
10:32:06.214103: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64'
10:32:06.214126: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64'
10:32:06.214149: pressure-vessel-wrap[1131709]: 	'--ro-bind'
10:32:06.214174: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64/libVkLayer_steam_fossilize.so'
10:32:06.214198: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64/libVkLayer_steam_fossilize.so'
10:32:06.214222: pressure-vessel-wrap[1131709]: 	'--ro-bind'
10:32:06.214246: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64/steamoverlayvulkanlayer.so'
10:32:06.214271: pressure-vessel-wrap[1131709]: 	'/games/bl3_steam/root/ubuntu12_64/steamoverlayvulkanlayer.so'

That all looks perfectly reasonable. And you say /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3 is a symlink to /games/bl3_steam/userdir.

Your strace when you were not mounting /games is consistent with that:

Documents/My Games", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3", {st_mode=S_IFLNK|0777, st_size=24, ...}) = 0
[pid 322337] lstat("/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved", 0x1c59e880) = -1 ENOENT (No such file or directory)

This says My Games is a real directory, My Games/Borderlands 3 is a symlink, and My Games/Borderlands 3/Saved doesn't exist - presumably because it resolves to /games/bl3_steam/userdir/Saved which, at the time, didn't exist.

However, now that you are mounting /games, inside the container, we should be able to find /games/bl3_steam/userdir perfectly well, because it's within the scope of the /games mount point.

What other symlinks are there in the path /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved, and what do they point to?

In a typical use of Proton, I would expect that at least c: is a symlink, usually to ../drive_c.

It might help you to figure this out if you run Steam with the additional environment variable PRESSURE_VESSEL_SHELL=instead, like this:

PRESSURE_VESSEL_SHELL=instead STEAM_COMPAT_MOUNTS=/games STEAM_LINUX_RUNTIME_LOG=1 PROTON_LOG=1 PRESSURE_VESSEL_VERBOSE=1 steam

Then try to launch the game. Instead of really launching the game, this will launch an xterm inside the container; it's a relatively minimal environment, so some of the tools you would expect might be missing, but it has all the basics like coreutils. Inside that xterm, you can poke around interactively and check that things are how you expect them to be inside /games. In particular, you can try the realpath command on various important/interesting paths, and see what they expand to.

If you want to run the game from there, you can cd back to the directory where you started (usually the game's installation directory) and run "$@" (with the quotes!) which is pre-set by pressure-vessel to expand to the command that it would otherwise have run.

One thing that could potentially be a problem is that some of the bind-mounts we set up are redundant, because we are already bind-mounting an ancestor of a directory (recursively), which means that bind-mounting a descendant with the same mode is pointless. However, I don't think that should be causing problems?

[smcv]

On a semi-related note, I'd been meaning to submit something to suggest a feature request that maybe the wineroot's My Documents should default to some symlink inside the user's homedir

That would be a feature request for Proton or Steam, rather than for pressure-vessel. Proton and Steam do most of the policy, we just do mechanism (although anything involving symlinks to another location causes extra work for pressure-vessel, because we have to know what to mount where).

However, I think it's deliberate that Proton makes each wineprefix self-contained, similar to winetricks sandbox: the games are intended to be somewhat insulated from each other, to minimize side-effects by one game on another, and I suspect a lot of people would be angry about Windows games leaving visible droppings at a (presumably) somewhat hard-coded path in their home directory, particularly if the games have cloud-sync. So that feature request might well be resolved "won't fix".

[apocalyptech]

Thanks for the info! Fun stuff, especially that xterm shell mode. On to answering some questions:

What other symlinks are there in the path /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved, and what do they point to?

In a typical use of Proton, I would expect that at least c: is a symlink, usually to ../drive_c.

Yep, apart from Borderlands 3, c: is the only symlink, which does go to ../drive_c:

drwxr-xr-x 94 pez pez 12288 Jan  8 15:28 /games
drwxr-xr-x 5 bl3 bl3 4096 Jan  8 23:57 /games/bl3_steam
drwxr-xr-x 3 bl3 bl3 4096 Jan 18 14:21 /games/bl3_steam/games
drwxrwxr-x 7 bl3 bl3 4096 Jan 18 10:24 /games/bl3_steam/games/steamapps
drwxrwxr-x 5 bl3 bl3 4096 Jan 18 12:28 /games/bl3_steam/games/steamapps/compatdata
drwxrwxr-x 3 bl3 bl3 4096 Jan 18 10:24 /games/bl3_steam/games/steamapps/compatdata/397540
drwxrwxr-x 4 bl3 bl3 4096 Jan 18 14:25 /games/bl3_steam/games/steamapps/compatdata/397540/pfx
drwxrwxr-x 2 bl3 bl3 4096 Jan 18 14:24 /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices
lrwxrwxrwx 1 bl3 bl3 10 Jan 18 10:24 /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c: -> ../drive_c
drwxrwxr-x 4 bl3 bl3 4096 Jan 18 10:24 /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users
drwxrwxr-x 24 bl3 bl3 4096 Jan 18 10:24 /games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser
drwxrwxr-x 8 bl3 bl3 4096 Jan 18 10:26 '/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents'
drwxrwxr-x 2 bl3 bl3 4096 Jan 18 10:29 '/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games'
lrwxrwxrwx 1 bl3 bl3 24 Jan 18 10:29 '/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3' -> /games/bl3_steam/userdir
drwxrwxr-x 6 bl3 bl3 4096 May  4  2020 '/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved'

As for running from that shell, everything appears to work just fine from there. Running commands in there always pops up the line ERROR: ld.so: object '/games/bl3_steam/root/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. -- I've trimmed that out of the output here, for readability.

Heading into savegame dir to check contents and realpath:

(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/common/Borderlands 3$ cd '/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved/SaveGames/76561197982540582'
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved/SaveGames/76561197982540582$ ls
./   1.json  10.sav  12.sav  14.sav  2.json  5.json  com.txt                mod_testing_gear.txt  profile.sav
../  1.sav   11.sav  13.sav  15.sav  2.sav   5.sav   make_testing_char.py*  peashooter.txt
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved/SaveGames/76561197982540582$ realpath .
/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582

Running the game, back to the dir we started from:

(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/compatdata/397540/pfx/dosdevices/c:/users/steamuser/My Documents/My Games/Borderlands 3/Saved/SaveGames/76561197982540582$ cd /games/bl3_steam/games/steamapps/common/Borderlands\ 3/
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/common/Borderlands 3$ "$@"

Back to the savegame dir to check out contents:

(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/games/steamapps/common/Borderlands 3$ cd /games/bl3_steam/userdir/Saved/SaveGames/76561197982540582
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ ls -l 10.*
-rw-rw-r-- 1 bl3 bl3 698085 Jan  4 00:27 10.sav
-rw-rw-r-- 1 bl3 bl3 698085 Jan 18 14:27 10.tmp
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ mv 10.tmp 10.sav
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ ls -l 10.*
-rw-rw-r-- 1 bl3 bl3 698085 Jan 18 14:27 10.sav
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ ls -l profile.*
-rw-rw-r-- 1 bl3 bl3 149117 Jan  4 00:27 profile.sav
-rw-rw-r-- 1 bl3 bl3 149117 Jan 18 14:27 profile.tmp
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ mv profile.tmp profile.sav
(steamrt soldier 0.20210105.2)bl3@arrakis:/games/bl3_steam/userdir/Saved/SaveGames/76561197982540582$ ls -l profile.*
-rw-rw-r-- 1 bl3 bl3 149117 Jan 18 14:27 profile.sav

So the game's read my savefiles just fine -- I can load up a save and do stuff -- but every time it tries to actually save out the game, it seemingly gets as far as writing a *.tmp file but then chokes before it can move it over the "real" file path.

I tried getting some more useful info with strace but was mostly stymied. Even allowing strace to capture all calls, I never actually see the creation of 10.tmp or profile.tmp (or any of the randomly-named tmpfiles in the drive_c root)... I get a bunch of lstat() attempts which fail, and then a few which return 0 since the file's been created in the meantime, but apparently that's not being captured. I wonder if that call happens in some other Wine process that's "above" Borderlands3.exe in the process tree, or something, and thus strace isn't finding it? I'd tried actually launching BL3 with strace -t -f %command% as the Steam launch options, but that never seemed to actually follow the process forks at all, despite having -f in there. (My other strace attempts were just attaching to the process after it had already launched.)

However, I think it's deliberate that Proton makes each wineprefix self-contained

Yeah, I'd sort of assumed that was probably by design, hence why I've still not gotten the impetus together to actually put it in as an RFE. :)

[smcv]

Taking a step back from the specifics of what @apocalyptech is doing, I think we probably have more than one root cause here.

@xDShot's report was lacking detail, so we don't know for sure, but I suspect it was the much simpler case that can be solved with STEAM_COMPAT_MOUNTS.

@apocalyptech seems to be going quite far beyond that, with a game-specific(?) user account, home directory and Steam installation that involves lots of symlinks: according to the SLR log, the "official" home directory for this user is /home/bl3 but the Steam installation is physically (after chasing symlinks) located in /games/bl3_steam/root.

I think we have to treat @apocalyptech's situation as unsupported, because each layer of symlinks introduces another place where things can go wrong. I can give some pointers for how to investigate, but if you've deliberately set up a situation this complex, it seems unlikely that I'll be able to reproduce the same setup.

Ironically, the reason that pressure-vessel does not indiscriminately mount as much of the host system as possible (which would reduce or remove the need for STEAM_COMPAT_MOUNTS) is that one day we want to be able to give each Steam game its own mock home directory, similar to the way most Flatpak apps work - which is a ligher-weight version of what @apocalyptech seems to be doing with each Steam game (or maybe just Borderlands 3?) having its own home directory, Unix user and Steam installation. So I think maybe we have similar long-term goals, but the specifics of how they're implemented might be fighting.

[smcv]

I tried getting some more useful info with strace but was mostly stymied

strace and the container are probably not going to work very well together.

One thing that might help is to use the beta branch of SteamLinuxRuntime_soldier, which starts up in a different way so that the launch options apply to more of it. It's the same procedure as opting in to the beta branch of a game (like CS:GO in https://support.steampowered.com/kb_article.php?ref=9847-WHXC-7326) but instead of changing the properties for a game, you'd go into the properties of the Steam Linux Runtime - soldier "tool" in your Steam library.

Another way to get more info might be to use the soldier SDK (which has its own copy of strace) instead of the normal Platform runtime, then use PRESSURE_VESSEL_SHELL=instead to get a shell, and run strace inside that shell. At the moment we don't have a particularly easy mechanism to switch to the SDK instead of the Platform, but something like this should work (at your own risk, changes will only last until the next Steam Linux Runtime update):

• download https://repo.steampowered.com/steamrt-images-soldier/snapshots/latest-container-runtime-public-beta/com.valvesoftware.SteamRuntime.Sdk-amd64,i386-soldier-runtime.tar.gz
• unpack it as SteamLinuxRuntime_soldier/soldier_sdk, so you have files like SteamLinuxRuntime_soldier/soldier_sdk/files/bin/env and SteamLinuxRuntime_soldier/soldier_sdk/metadata
• edit the _v2-entry-point shell script and look for the call to run-in-steamrt
• remove --deploy from its command-line arguments
• replace ${deploy} with soldier_sdk

One day I want to move more of this logic into pressure-vessel so that it's more "joined up", but there's a lot of stuff that had to be bolted on to handle Proton game startup happening in multiple steps, hence the several layers of shell script wrappers.

[smcv]

or any of the randomly-named tmpfiles in the drive_c root

Hmm, this might be important. I wonder whether something (Proton? Wine? wineserver? Borderlands 3?) is assuming that all of drive_c is one monolithic mount-point, such that rename() and link() will work between dissimilar directories below drive_c?

That assumption would normally be trivially true, but because you've made My Documents/My Games/Borderlands 3 a symlink, it doesn't necessarily hold true any more. The only place it's possible to use rename() and link() reliably, in the presence of symlinks, is within a single directory - so if something is doing the standard POSIX atomic-overwrite-via-rename trick, it would be OK to do that within a directory, but it is not reliable to do it between drive_c and drive_c/.../Borderlands 3/Saves/....

[smcv]

I also see that 397540/pfx/dosdevices/c: has a bunch of temp files being created in it, like ... rf6ef5.tmp

Looking at Wine source code, this looks suspiciously like a call to GetTempFileNameW() with path = C:\ and prefix = rf, which in turn looks like it's probably a call to ReplaceFileW().

So I think my guess might be right: Borderlands 3 is (quite reasonably) assuming that it can create a temp file in C:\ and then rename it to C:\...\Borderlands 3\Saves\..., because everything on drive C: is presumably on the same filesystem - but your use of symlinks has made this rename actually be crossing between mount points (where it would ordinarily not), so it fails.

If pressure-vessel knew how to eliminate the redundant bind-mounts then that might resolve this. However, I don't think we can make this a high priority, because the only reason this is happening is that you've replaced part of the WINEPREFIX with a symlink - and if it had been a symlink to a location that did not share a common parent, there'd be nothing that pressure-vessel could do to solve that.

Doing a bit more strace action I can see that the game's looking for these files but apparently not finding them

If I'm right about this being GetTempFileNameW(), then that might well be working as designed: Wine is generating a random name for a temp file, then probing the filesystem to check that a temp file of that name doesn't already exist (if it did, Wine would pick a different name).

[apocalyptech]

I think we have to treat @apocalyptech's situation as unsupported, because each layer of symlinks introduces another place where things can go wrong. I can give some pointers for how to investigate, but if you've deliberately set up a situation this complex, it seems unlikely that I'll be able to reproduce the same setup.

Heh, yeah, fair enough. I realize I'm probably an edge case for this kind of stuff. :)

@apocalyptech seems to be going quite far beyond that, with a game-specific(?) user account, home directory and Steam installation that involves lots of symlinks: according to the SLR log, the "official" home directory for this user is /home/bl3 but the Steam installation is physically (after chasing symlinks) located in /games/bl3_steam/root.

BL3's actually the only game where I've got a separate user set up for it, related to some modding activity I've been doing on the game; it was useful to be able to selectively mark + route packets coming from BL3, and iptables' --user method is about the most straightforward way to do that. It's definitely not something I do habitually.

I do have my Steam install root physically separate from /home even on my main account, though -- I NFS-mount /home from a NAS on my network, which gets shared between various machines I log in on. It's much nicer to have stuff like Steam on local disk, though, so my Steam install roots are all in /games.

One thing that might help is to use the beta branch of SteamLinuxRuntime_soldier... Another way to get more info might be to use the soldier SDK (which has its own copy of strace) instead of the normal Platform runtime

Cool, thanks for those ideas; I'll give 'em a go in a bit!

[apocalyptech]

So I think my guess might be right: Borderlands 3 is (quite reasonably) assuming that it can create a temp file in C:\ and then rename it to C:\...\Borderlands 3\Saves\..., because everything on drive C: is presumably on the same filesystem - but your use of symlinks has made this rename actually be crossing between mount points (where it would ordinarily not), so it fails.

Ahh, interesting, that does sound totally reasonable (well, apart from something trying to create a tmpfile right in C:\ to begin with, but what're you gonna do about that, eh?).

I'll have to see how it deals with having a bind mount instead of a symlink -- if that doesn't do the trick then I'll just cope with leaving those saves inside compatdata/<gameid>. Thanks again for all your help and insights, appreciate it!

[smcv]

Starting from pressure-vessel version 0.20220803.0 in today's "Steam Linux Runtime - soldier" and "Steam Linux Runtime - sniper" betas, more top-level directories are available to games by default, reducing the need to use STEAM_COMPAT_MOUNTS. The affected paths are the typical paths for removable media:

• /media: FHS parent directory for removable media mount points
• /mnt: FHS traditional/legacy temporary mount point
• /run/media: widely-used alternative to /media

and some FHS locations where people often put the mount points for non-removable but non-OS drives and partitions, such as a secondary SSD or HDD:

• /home: FHS parent directory for home directories
• /opt: FHS directory for self-contained third-party software
• /srv: FHS directory for site-specific services' data

If symbolic links in a directory used by the game point into one of these locations, then the target of the symlink will be visible to the game, even if it wasn't previously. This means those symlinks will generally work.

Custom top-level directories like the /games mentioned in this issue are not shared by default, and still require use of STEAM_COMPAT_MOUNTS, because we can't automatically classify unknown top-level directories into things you would expect to be shared (/storage, /games, /large-disk-drive) and things you would not expect to be shared (future OS components, /lost+found, /corporate-secrets). Technical details

In the absence of more information from the original issue reporter @xDShot, I'm assuming that this will mostly solve the issue as initially reported.

[smcv]

There is a more specific sub-issue that @apocalyptech described, which I'll summarize here:

• a game expects to be able to create a temporary file in C:\ and then rename it into C:\...\Borderlands 3\Saves\...
• one of the components of that path (let's say Borderlands 3) is actually a symlink pointing into a different mount point
• the rename() fails, because renames can't cross between mount points

I don't think this specific sub-issue can be solved, and I suspect that if it was using Windows 10 symlinks, the same thing might well happen on native Windows. I would recommend treating the Wine/Proton drive_c directory as indivisible/atomic, and not putting symlinks or bind-mounts inside it.

[smcv]

Starting from pressure-vessel version 0.20220803.0 in today's "Steam Linux Runtime - soldier" and "Steam Linux Runtime - sniper" betas, more top-level directories are available to games by default, reducing the need to use STEAM_COMPAT_MOUNTS.

This change was promoted from beta to the default branch today.

[smcv]

In the absence of more information from the original issue reporter @xDShot, I'm assuming that this will mostly solve the issue as initially reported.

@kisak-valve, I think it's time we close this issue: everything that is solvable here has been solved.