olmod.sh steam deck (gamescope) compatability

[kevenwyld]

olmod.sh seems to not work correctly when attempting to launch through the default steam deck UI. I believe this is due to the steam UI running inside a gamescope session. I've been able to reproduce this behavior on the desktop as well using gamescope directly.

Apologies this is going to get a little into the weeds because I had to work around some quoting issues in launch options as well. I'll try to explain them in as much detail as I can.

Steps to reproduce:

1. Launch steam in a gamescope session.
   I do this with a little script below. This is basically what the deck does however the deck assigns gamescope to a TTY directly with -O *,eDP-1

] > cat bin/steam-gamescope-16x9.sh
#!/bin/sh
# See https://developer.valvesoftware.com/wiki/Command_Line_Options
/usr/bin/gamescope -W 2560 -H 1440 -r 165 -b --steam -- /usr/bin/steam -tenfoot

2. Install olmod using the steps in the wiki for linux
3. Configure launch options as follows, working around the quoting issue by specifying the LD_PRELOAD path manually instead of using the script.

OLMODDIR=$HOME/.steam/root/steamapps/common/Overload LD_PRELOAD=$HOME/.steam/root/steamapps/common/Overload/olmod.so $HOME
/.steam/root/steamapps/common/Overload/Overload.x86_64 > olmod_debug.log 2>&1; echo "%command%"

• This is because on the deck LD_PRELOAD already has the gameoverlayrenderer.so path in it and using that and olmod.so at the same time causes a segfault. Surprisingly even disabling the overlay in the game settings still populates these libraries in the LD_PRELOAD path.
• The %command% at the end is because steam assumes that if %command% is missing from the launch options everything you put there is an argument that should follow the normal launch command, so putting it at the end launches an alternative executable and isolates all the other things that steam wants to put on the command line.
• I found that on my desktop which runs swaywm (wayland) it was necessary to also add the SDL_VIDEODRIVER=x11 env var to get the game to draw a window correctly. Otherwise it would run in the background using a lot of CPU resources but not display anything.

4. Run the game by pressing the play button. On both the deck and the desktop this results in the spinning valve icon but no game window. It is running in the background (verified by grepping the process tree) but nothing is drawn.

Log output from the debug shell redirect above:

(deck@steamdeck Overload)$ cat olmod_debug.log
Found path: /home/deck/.local/share/Steam/steamapps/common/Overload/Overload.x86_64
Mono path[0] = '/home/deck/.local/share/Steam/steamapps/common/Overload/Overload_Data/Managed'
Mono config path = '/home/deck/.local/share/Steam/steamapps/common/Overload/Overload_Data/Mono/etc'
Preloaded 'ScreenSelector.so'
Preloaded 'libCSteamworks.so'
Preloaded 'libopenvr_api.so'
Preloaded 'libsteam_api.so'
Unable to preload the following plugins:
	libUnetEncryption.so
	ScreenSelector.so
	libCSteamworks.so
	libsteam_api.so

Logging to /home/deck/.config/unity3d/Revival/Overload/Player.log

Log output from the Player.log file:
https://gist.github.com/kevenwyld/44bfca40c48b66fdb426f1dc58c3666e

When launching from desktop mode without changing any of the options listed above everything works fine. It also works fine on desktop if I launch steam outside of gamescope.

This is all to say that I'm stuck at this point but I have the hardware, so if someone comes along and has a better understanding of how the mod works I'd be happy to help any way that I can. It would be really cool to be able to use the mod just like any other game on the deck instead of having to switch to desktop mode. This has other advantages like a functional mouse pointer and the overlay.

Also, and most importantly, thank you everyone for this rather amazing achievement. I am in awe of the accomplishments of this community.

[derhass]

Hmm, I tried to have a look at this. Problems are: I don't have steam, I don't have a steam deck, and I don't use wayland.

However, I grabbed the latest release of winesapos, put it on an USB pen drive, and created an empty steam account.

I got olmod running, from steam, while running in a gamescope session. What I did was just modifying olmod.sh, line 32 from

OLMODDIR="${olmoddir}" LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}./olmod.so" "${overload}" ${olargs}

to

OLMODDIR="${olmoddir}" LD_PRELOAD="./olmod.so" "${overload}" ${olargs}

and adding that modified olmod.sh as a non-steam game to the steam library. I did not try to fiddle around with any launch parameter. (And I have no idea what quoting issue you are referring to - the script uses proper double quoting where needed.)

@kevenwyld Can you please try to check if that approach still fails for you?

[kevenwyld]

@derhass thanks for taking the time to test this. So initially your suggestion didn't work on the deck, however it seems that when olmod is launched the deck focuses the official game. Probably for the same reason that you still see "so-and-so is playing Overload" even when they are using olmod outside of steam. You can hear the menu music but can't see the game. What I discovered is that you have to go to the steam menu on the left and select the non-steam olmod again, then it works fine. See the video below (the startup takes a bit, sorry):

VID_20230806_095445_smaller.mp4

Your way, including editing the script, is a lot better than what I was doing. Thanks for the help! I guess I was trying to preserve the script so future upgrades didn't break anything, but that's probably not as important as I made it in my head. Same with the LD_PRELOAD quoting problem which I'm pretty sure I imagined and it doesn't exist =] .

The situation is the same on the desktop when running in big picture, but it's easier to work around using the desktop UI there than it is on the deck.

[derhass]

Hmm. Interesting. Looks like you at least have some workaround, even though it's still clumsy. I still have no idea what's going on on the Deck, seems to be specific to the Steam OS and/or hardware used there, at least I have no idea how I could reproduce this behavior.

As for the script: You could simply create your own renamed copy of that script so it won't get overwritten when you update olmod.

The problem with the script in its default state is that steam's gameoverlayrenderer.so as well as olmod.so hook the dlsym function itself - and get into each others way when both are present in the same process. If I leave the script as default, this leads to olmod being added later, and being completely ignored. If I change the script to add olmod first, it crashes.

[kevenwyld]

I still have no idea what's going on on the Deck, seems to be specific to the Steam OS and/or hardware used there, at least I have no idea how I could reproduce this behavior.

I think it's probably out of scope for this project to deal with that issue. This workaround is more than fine in comparison to switching to desktop mode just to play this game.

Thanks again!

[derhass]

Well, it would be nice if olmod would work more out-of-the-box on the steam deck. But for now, I don't have the ability to investigate this any further...