Add support for forwarding commands into the container environment #5891
Conversation
| rc = self.run_proc([g_proton.wine64_bin, "c:\\windows\\system32\\steam.exe"] + sys.argv[2:] + self.cmdlineappend) | ||
| argv = [g_proton.wine64_bin, "c:\\windows\\system32\\steam.exe"] | ||
|
|
||
| rc = self.run_proc(adverb + argv + sys.argv[2:] + self.cmdlineappend) |
There was a problem hiding this comment.
Not really related to this feature at all, but if we don't have a remote debug process, it might be nicer if the proton script did an execve() to completely replace itself with the wine64 ... command?
There was a problem hiding this comment.
I'll have a look into this. Sounds sensible, but I wonder if there's a reason to have it like that. Only thing that comes to mind is to have proton easily identifiable in the process tree.
|
I don't think I fully understand how we ended up with this design. Why can't one just execute Also I'm not sure if I understand the intent here correctly or how it makes this easier. Correct me if I'm wrong but the intended use case seems to be:
We may want to add a view more verbs to |
It's a container environment (analogous to Flatpak or Docker or whatever), so setting environment variables isn't sufficient to "enter" it. To see the same libraries that Proton depends on, and more generally the same view of the filesystem that Proton has, you can't just run commands on the host system. Instead, we need an IPC protocol with a server inside the SRT environment, and a client outside that sends commands to the server: analogous to having an Because this is all local to one machine, we don't need the encryption or networking that The alternative to using IPC would be to do complicated things with namespaces to force a process into the container from the outside. |
The Steam container runtime doesn't know anything about Proton: it's just a generic library stack, and doesn't know whether the command it is going to be running is Proton or a native Linux game. It's only the Proton scripts that know how Proton/Wine works, and therefore how to get a command running in the desired execution environment.
It could be if you wanted it to be, but it would have to be the Proton startup script that did this, because the Steam container runtime does not know anything about Proton.
Again, the container runtime is not in a position to do this, because it has no idea what its "payload" is: it could be a native Linux game, it could be Proton, it could be dosbox, ... |
Part of the idea of this command-injection interface was that the compatibility tool developer doesn't have to anticipate everything that the game developer could possibly want: if the debugging entry point is "run an arbitrary command", then that command can be anything that exists in the container. Things like For the diagnostic tools that are included in Wine/Proton, the container runtime has no idea what Proton is or where to find it, but Proton does know that, and is in a position to add For debugging Windows games specifically, it seems like there are two things you could reasonably want to do: one is to run some Linux program like |
|
@ivyl - sorry this probably comes very out of context. The internal task is https://gitlab.steamos.cloud/jupiter/tasks/-/issues/573, which CW should have access to. @nsivov will have more context from the initial email thread with Valve. This particular patch is from the prerequisite work in https://gitlab.steamos.cloud/jupiter/tasks/-/issues/483. I have been able to use an earlier version of this patch to trigger the
Having the launcher service below the proton script in the hierarchy is working well. As Simon pointed out, additional verbs would impair our ability to easily extend the functionality, and also requires ad-hoc knowledge in the Steam client which I'm trying to avoid for the same extensibility reasons. |
I get this. The question is what's the benefit of
True. If we go this way I think this is something we would need to add.
I see Steam as proton / compatdata manager and Proton mostly as a clueless runner. To me it feels like setting
Sounds like something that would be solved via
I think adding wine's bin/ to PATH won't hurt in general. I don't think it would mess with games if we would do that unconditionally.
It continues to eludes me why we want to add all that complexity and inscribe all the runtime implementation details into I'm still trying to understand why Thanks for the context. I believe that with
I see. How about having a generic verb that just executes anything with Proton's environment? I'm just not a fan of putting all the knowledge about how the runtime works into |
Steam doesn't really know what Proton is or what it does, though; it has a generic concept of compatibility tools, but it doesn't know the difference between Proton (which runs Windows executables on Linux), SLR (which runs Linux executables on Linux), or a hypothetical dosbox compatibility tool (which runs DOS executables on Linux). All Steam knows is that there is a compatibility tool that it has been configured to run the game under (in this case Proton), and that because that tool has a dependency in its When we run native Linux games under
That would require Steam to have domain-specific knowledge about Proton being special (or more generically, a configuration parameter that basically has the semantics of "this is Proton and is special"), which is something we're trying to avoid. I thought a lot of the purpose of Proton was that it encapsulates the domain-specific knowledge that is involved in using (a suitably modified version of) Wine as a Steam compat tool? (As opposed to what might go in a different compat tool that used Dosbox or ScummVM or something.)
I'm treating the IPC server and the If we were only dealing with native Linux processes, then I would have suggested that Steam should treat the But that doesn't work in the presence of Proton (or other compat tools that translate between incompatible calling conventions, like dosbox), because the calling convention for a Windows-onto-Linux compat tool like Proton is that it expects a path to a Windows executable as its parameter, not a path to a Linux executable. I'm also reluctant to suggest having the "API" involve Steam setting something like
Right, that would be better than having However, this would mean the If we run the IPC server inside the last compat tool in the chain (in this case Proton), then it is a child of the compat tool, so it automatically inherits all the inheritable state from the compat tool, including future kernel concepts that haven't even been invented yet. That seems like a good thing to me. The last one in the chain also makes sense to me as being the special case, because there is already something special about it: it's the last one. Another interesting question for this approach would be: who runs Proton with that verb, how, and in what environment? If it works the same as Meanwhile, it's not enough for Steam to run the So now someone (Steam?) needs to know at what point to "cut" the chain of compat tools at one that supports the IPC server interface, use the IPC client/server pair to replace that point and everything before it, but then find the Proton script among the rest and run it in the environment that the IPC server provides? |
|
I generally agree with Simon's comments above. In particular that neither the compat tool(s) above Proton, nor Steam itself have knowledge of what they are running and we'd like to keep it that way. I also understand wanting to keep Proton generally agnostic to being run within a compat tool environment vs not (a local dev host build for instance). It would be cleaner. Maybe one option is that the compat tool version has a different I don't think this extra complexity is very justified though, and I'm concerned about re-executing |
|
Thank you both for detailed explanations. This has changed my view on the place of Proton in the full ecosystem. Indeed, if you aim for having more compatibility tools than just SLR and Proton it makes sense for Steam to not have detailed knowledge about them. Also I have few general questions that still bog me:
|
You're right that this is a bit odd, but the I inserted it at the last point in the chain where we're still in Linux-world, just before handing over to Windows-world, so that it's as close to the actual game's execution environment as it can be. An advantage of this is that when poking in commands from the outside, the interpretation is consistent: if you're currently running Steam game 440 (which happens to be TF2), you can always run If we were using a Windows executable as the IPC service, then it would be impossible to use the same implementation we use for Linux games, and we also wouldn't be able to use fd-passing to hook up stdin/stdout/stderr transparently. The next best thing would be something TCP-based like ssh, which means dealing with authentication and security; and the user of the IPC client would need to know whether it was expecting to be told to run a Linux process like If Wine had more knowledge of the inner workings of the IPC server, then in principle I suppose we could give it a way to specify "the thing I want to run is a Windows executable": and have it automatically turn that into |
I have wondered about this myself, but I think we should probably iterate a bit to make it as useful as possible, and then introduce a convenience entry point for "do the right thing" once we are more confident that we know what the right thing is. If we're running a wrapper unconditionally, then it's a single point of failure, and crash bugs in it become critical (preventing running any games, not just for developers but also for users), which does concern me. Creating an "adverb" process like this is reasonably cheap in terms of startup times, but not free, which is why you see the generically-named (I did consider combining the |
I basically agree, but how would that work? It's easy for Steam to set environment variables that apply to the entire subprocess hierarchy, but hard to set them on a per-tool basis so that SLR_soldier and Proton each see their own correct app-ID. The least-bad thing I can think of is to insert a bunch of which seems quite nasty, and will be problematic for anything that parses the command-line and tries to pick it apart into compat tools (which nobody should be doing in production, but the Using Also, if one of the executable names might have an equals sign in it (for example if a user's Steam library is (Arguably it should be |
I think what we are doing here is the most sensible option for developers, even though it creates a bit of dissonance and requires knowledge about Proton/Wine.
It would be still better to have the entry point already so we can do something like
If this is too scary to call it unconditionally then we can hide it behind an extra check of
Then maybe we should not assume that the app knows the ID and instead we can rely solely on the human-friendly names like |
diff --git a/proton b/proton
index 4bf158b5..06629194 100755
--- a/proton
+++ b/proton
@@ -1450,9 +1484,14 @@ class Session:
return subprocess.call(args, env=local_env, stderr=self.log_file, stdout=self.log_file)
def run(self):
+ if shutil.which('steam-runtime-launcher-service') is not None:
+ adverb = ['steam-runtime-launcher-service', '--tool-name', 'proton', '--' ]
+ else:
+ adverb = []
+
if "PROTON_DUMP_DEBUG_COMMANDS" in self.env and nonzero(self.env["PROTON_DUMP_DEBUG_COMMANDS"]):
try:
@@ -1467,9 +1506,11 @@ class Session:
# CoD: Black Ops 3 workaround
if os.environ.get("SteamGameId", 0) == "311210":
- rc = self.run_proc([g_proton.wine_bin, "c:\\Program Files (x86)\\Steam\\steam.exe"] + sys.argv[2:] + self.cmdlineappend)
+ argv = [g_proton.wine_bin, "c:\\Program Files (x86)\\Steam\\steam.exe"]
else:
- rc = self.run_proc([g_proton.wine64_bin, "c:\\windows\\system32\\steam.exe"] + sys.argv[2:] + self.cmdlineappend)
+ argv = [g_proton.wine64_bin, "c:\\windows\\system32\\steam.exe"]
+
+ rc = self.run_proc(adverb + argv + sys.argv[2:] + self.cmdlineappend)
if remote_debug_proc:
remote_debug_proc.kill()
diff --git a/toolmanifest_runtime.vdf b/toolmanifest_runtime.vdf
index 0b8490e6..8c32fcb1 100644
--- a/toolmanifest_runtime.vdf
+++ b/toolmanifest_runtime.vdf
@@ -1,6 +1,7 @@
"manifest"
{
"version" "2"
+ "name" "proton"
"commandline" "/proton %verb%"
"require_tool_appid" "1391110"
"use_sessions" "1"
This is more along the lines what I would like to see. We don't need to know our app id and instead Steam / other tooling can discover our developer-friendly name via toolmanifest. Also here If it's too scary we can add |
I'll have to defer to @TTimo on whether that's feasible within Steam, I don't have access to Steam itself. |
If SRT and Proton are invoking something unconditionally, I don't really want the thing they invoke to be steam-runtime-launcher-service, because it has non-trivial code and dependencies, which could easily make all SLR and Proton games slow or non-functional; and because that isn't a versioned name, which will give us problems if we want an incompatible behaviour or interface change. Having a more trivial entry point with a versioned name like I suspect we'll go through a few interface versions initially, because I'll have to guess how it should behave with only a limited ability to prove that it works for real-world use-cases, but perhaps I'm being pessimistic. |
Fingers crossed. That would make our life much easier with the many, many apps we have for Proton and the occasional manual build :-)
Sounds like a not too bad of an option. I imagine that we would need to pass the tool name somehow, either as a environment variable, like you have suggested, or the first parameter. diff --git a/proton b/proton
index 4bf158b5..06629194 100755
--- a/proton
+++ b/proton
@@ -1450,9 +1484,14 @@ class Session:
return subprocess.call(args, env=local_env, stderr=self.log_file, stdout=self.log_file)
def run(self):
+ if shutil.which('steam-runtime-launcher-interface-0') is not None:
+ adverb = ['steam-runtime-launcher-service', 'proton']
+ else:
+ adverb = []
+
I'm not completely opposed to keeping Out of curiosity, why does the bus name include the SteamAppId of the game? |
I meant more like |
It's not clear to me whether Steam can run more than one game at a time; and if it can't, it's not clear to me whether that's considered to be a bug that might be fixed in a future Steam release. The bus name being non-unique is not necessarily a fatal problem with the latest steam-runtime-tools release, which has a |
Having a human friendly / common name carried in the manifest should be easy. I've read through the rest of the discussion and I generally don't have concerns. Please pick a design (using adverbs |
Great! So we can just add the "name" field? The toolmanifest has "version" field and I wonder if we need to touch that or not.
I'll cook up a version using this. |
|
@smcv I think you'll need to do similar changes in SLR? |
Yes, and I'll also need to write the versioned entry-point. I'll hopefully start on that tomorrow. |
Recent versions of the Steam Runtime include an IPC server/client pair which can be used to run commands inside the container environment (or any other special execution environment), analogous to sshd/ssh or flatpak-portal/flatpak-spawn. The server runs inside the Steam Runtime container and accepts commands over D-Bus; the client runs on the host system, asks the server to run a command, and forwards its stdin, stdout and stderr back to the host. https://gitlab.steamos.cloud/steamrt/steamlinuxruntime/-/merge_requests/72 adds support for injecting commands into the SteamLinuxRuntime_soldier compatibility tool (and any later version, such as sniper). However, Steam compatibility tools are stackable: in particular, Proton runs in a soldier container (or presumably sniper in future). If we are debugging a Proton game, then ideally we will want to inject commands into Proton's execution environment rather than soldier's, so that they run with the correct environment variables etc. to communicate with a running Proton session. In particular, it's important that the `WINEPREFIX` is correct. The steam-runtime-launcher-interface-0 program implements the interface for compatibility tools to use to decide where, if anywhere, to launch the command server. This commit does not alter the scripts produced by PROTON_DUMP_DEBUG_COMMANDS. To run those scripts' commands in the container environment, pass their filenames to steam-runtime-launch-client. Signed-off-by: Simon McVittie <smcv@collabora.com>
Implemented, and seems to work. You'll need a bit of juggling components to make it work before this stuff appears in public releases (see edited MR description), but surprisingly little of that is necessary. |
I think we should have a more descriptive field name here than just The semantics that we want for it are:
Having Proton-in-Proton would make no sense, so all Protons can safely have the same token here, and nesting containers doesn't work, so all container runtimes can safely have the same token here. I'm not sure what expresses that best. Perhaps something involving |
|
how about |
|
Thanks! I've added a commit that sets the compatmanager_layer_name. The PR is okay to merge, I'm happy about how it has shaped up :-) I think it should be safe to have it in bleeding-edge already. Let me know if you want it merged now or wait a bit. As of the debug scripts - I still see value in having them dumping the whole custom env, e.g. to use with standalone wine. It still has some use for outside-of-SRT debugging that Proton developers do. I agree it's not perfect and it's a more of advanced topic. There's a long overdue cleanup done too. I'll handle that separately. As of the second |
|
Not directly related to this, but note that we'd like to do away with the toolmanifest vdf that is embedded in the compat tool files eventually. It's a remnant from early implementation and causes some minor problems, we'd like to consolidate that to the appinfo. |
For SLR (and probably Proton too) we'll need to be quite careful that the tool manifest stays in sync with the actual capabilities and behaviour of the compat tool, including across upgrades, and in situations where different branches have different (incompatible) behaviour. I know we've had some trouble with this in the past, where Steam updated one of the SLR depots to a version with incompatible behaviour, but didn't reload Or would the proposed change make Steam more able to synchronize its opinion on each compat tool with the reality, by having the tool manifest live somewhere that Steam can reload more regularly? |
|
It's now merged into our staging experimental_7.0 branch and is already live on experimental's It will make it into the next release of experimental and 7.0 dash release. Thanks! |
|
@ivyl FYI, the necessary pressure-vessel version has also been released into the The requirement is that Steps to testRun Steam from a terminal with no special options or environment variables. Opt in to Proton Experimental Verify integrity (just to make sure no temporary hacks are still in place). Have any Proton game set to use Proton Experimental. I used The Expendabros (312990) which is small and free-to-play, so anyone can follow these steps. Put Launch the game. This text appears on Steam's stderr: (Obviously the Steam Library path /home/me/SteamLibrary and the unique name :1.430 will vary.) Run the indicated command in another terminal. You get an interactive shell inside the container. Its current working directory is the game's cwd (in my case
If the command printed on stderr is not convenient to retrieve (perhaps it's going to a log or the Journal or /dev/null), another way to get into the container is to run: and it will show you a list of For completeness: if you use ErrataAt the moment, if you want to be in the game's working directory, you have to use It's possible to use |
Recent versions of the Steam Runtime include an IPC server/client pair which can be used to run commands inside the container environment (or any other special execution environment), analogous to sshd/ssh or flatpak-portal/flatpak-spawn. The server runs inside the Steam Runtime container and accepts commands over D-Bus; the client runs on the host system, asks the server to run a command, and forwards its stdin, stdout and stderr back to the host. https://gitlab.steamos.cloud/steamrt/steamlinuxruntime/-/merge_requests/72 adds support for injecting commands into the SteamLinuxRuntime_soldier compatibility tool (and any later version, such as sniper). However, Steam compatibility tools are stackable: in particular, Proton runs in a soldier container (or presumably sniper in future). If we are debugging a Proton game, then ideally we will want to inject commands into Proton's execution environment rather than soldier's, so that they run with the correct environment variables etc. to communicate with a running Proton session. In particular, it's important that the `WINEPREFIX` is correct. The steam-runtime-launcher-interface-0 program implements the interface for compatibility tools to use to decide where, if anywhere, to launch the command server. This commit does not alter the scripts produced by PROTON_DUMP_DEBUG_COMMANDS. To run those scripts' commands in the container environment, pass their filenames to steam-runtime-launch-client. Signed-off-by: Simon McVittie <smcv@collabora.com> Link: #5891
Recent versions of the Steam Runtime include an IPC server/client pair which can be used to run commands inside the container environment (or any other special execution environment), analogous to sshd/ssh or flatpak-portal/flatpak-spawn. The server runs inside the Steam Runtime container and accepts commands over D-Bus; the client runs on the host system, asks the server to run a command, and forwards its stdin, stdout and stderr back to the host. https://gitlab.steamos.cloud/steamrt/steamlinuxruntime/-/merge_requests/72 adds support for injecting commands into the SteamLinuxRuntime_soldier compatibility tool (and any later version, such as sniper). However, Steam compatibility tools are stackable: in particular, Proton runs in a soldier container (or presumably sniper in future). If we are debugging a Proton game, then ideally we will want to inject commands into Proton's execution environment rather than soldier's, so that they run with the correct environment variables etc. to communicate with a running Proton session. In particular, it's important that the `WINEPREFIX` is correct. The steam-runtime-launcher-interface-0 program implements the interface for compatibility tools to use to decide where, if anywhere, to launch the command server. This commit does not alter the scripts produced by PROTON_DUMP_DEBUG_COMMANDS. To run those scripts' commands in the container environment, pass their filenames to steam-runtime-launch-client. Signed-off-by: Simon McVittie <smcv@collabora.com> Link: #5891
Developed against
proton_7.0-rcThis version requires https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/merge_requests/465, and doesn't alter the
PROTON_DUMP_DEBUG_COMMANDSscripts. Old version at https://github.com/smcv/Proton/tree/launcher-service1 for reference.To test with current versions of
SteamLinuxRuntime_soldierand Steam, some reordering of chickens and eggs is required:STEAM_COMPAT_LAUNCHER_SERVICE=protonin Steam's environment or in a game's Launch Options. This will stop being necessary when the work by @TTimo that this is intended to enable gets a bit further.SLR_soldier doesn't have the newSLR_soldiersteam-runtime-launcher-interface-0, so replaceSteamLinuxRuntime_soldier/pressure-vesselwith a build from https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/merge_requests/465. https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/jobs/95761/artifacts/raw/_build/production/pressure-vessel-bin.tar.gz should be suitable. This will stop being necessary whenlauncher-interface-0gets merged and released.client_betais suitable.The newSLR_soldiersteam-runtime-launcher-interface-0isn't inPATHyet, so setPRESSURE_VESSEL_SHELL=insteadin Steam's environment or in a game's Launch Options. When you start the game, instead of the actual game, you'll get anxtermrunning in the container. In thatxterm, useexport PATH=/run/pressure-vessel/pv-from-host/bin:$PATHto get the necessary tool into thePATH, then run"$@"(including the quotes) to run Proton and the actual game. This will stop being necessary whenlauncher-interface-0gets merged and released.client_betais suitable.https://gitlab.steamos.cloud/steamrt/steamlinuxruntime/-/merge_requests/76 is the corresponding glue to get this launched in the soldier/sniper container ((This has been integrated.)STEAM_COMPAT_LAUNCHER_SERVICE=container-runtime) or in the scout-on-soldier "layered" environment that can be used for native Linux games (STEAM_COMPAT_LAUNCHER_SERVICE=scout-in-container).proton: Allow forwarding commands into the Proton environment
Recent versions of the Steam Runtime include an IPC server/client pair
which can be used to run commands inside the container environment
(or any other special execution environment), analogous to sshd/ssh or
flatpak-portal/flatpak-spawn. The server runs inside the Steam Runtime
container and accepts commands over D-Bus; the client runs on the host
system, asks the server to run a command, and forwards its stdin, stdout
and stderr back to the host.
https://gitlab.steamos.cloud/steamrt/steamlinuxruntime/-/merge_requests/72
adds support for injecting commands into the SteamLinuxRuntime_soldier
compatibility tool (and any later version, such as sniper). However,
Steam compatibility tools are stackable: in particular, Proton runs in a
soldier container (or presumably sniper in future). If we are debugging
a Proton game, then ideally we will want to inject commands into Proton's
execution environment rather than soldier's, so that they run with the
correct environment variables etc. to communicate with a running Proton
session. In particular, it's important that the
WINEPREFIXis correct.The steam-runtime-launcher-interface-0 program implements the
interface for compatibility tools to use to decide where, if anywhere,
to launch the command server.
This commit does not alter the scripts produced by
PROTON_DUMP_DEBUG_COMMANDS. To run those scripts' commands in the
container environment, pass their filenames to
steam-runtime-launch-client.