Improve UX of the RivaTuner Statistics Server (RTSS) crash

[Madis0]

Request Description

Sodium recently added a forced crash when RivaTuner is present.

While it makes sense to have it, it is currently bad UX:

• the message may be instantly seen in some launchers (e.g. vanilla launcher) but not all (e.g. Modrinth launcher)
• the user may incorrectly assume Sodium itself/the modpack/any other mod is broken (without reading the logs)
• the user may go through a time-consuming process of troubleshooting potential issues/getting logs with a support community etc, while at the end the issue was much simpler and the only the user themselves can fix it
• the error in itself is not actually a bug as much as it is an incompatibility, so a visual GUI similar to Fabric/Quilt warning screens would make more sense
• some modpacks may be inclined to add the JVM flag just to improve the UX, which is obviously not what you would recommend, so a better alternative is needed

Therefore, I suggest adding a similar Java GUI warning like it was done in #2250.

[jellysquid3]

Someone else needs to write the code for that alternative, since I don't have a Windows computer and had to implement the existing solution blindly through the Win32 documentation.

Unfortunately, there are two problems currently:

• We can't use Java Swing code (like Add warning when the user double clicks the jar file like a program #2250 did) because it depends on Java's AWT library which LWJGL is incompatible with. That means once the game is loaded, it becomes impossible to use any existing GUI code.
• The toast system is only designed for temporary messages right now, and it can't render during early game startup. If a user is opening RivaTuner in the background, chances are they are not going to have Minecraft focused, so the toast message will never be seen.

There is a small abstraction over pop-up message boxes here that uses platform-specific API calls (again, written only using Win32 documentation, but people say it works). This is probably the best we can do, and it's at least specifically designed to show message boxes while the game is running.

[jellysquid3]

But it's worth mentioning that at the end of the day, the UX is not ever going to be good, because the game process has to crash once RTSS is injected. If we continue rendering, the user's hard drive will be thrashed, and that's even worse (in my opinion.)

I would have preferred to instead blacklist specific versions of RTSS, but like I said here it's not possible for us to determine which version of RTSS is loaded, because their developers do not provide any version information to other applications.

[jellysquid3]

I did some reverse engineering of how RTSS works, and since v7.3.0 it appears that if you set the environment variable RTSSHooksCompatibility=1 (not documented anywhere...) that will disable the hooking functionality. But the RTSS developers also have logic in there to specifically prevent certain applications from enabling this, and they seem to indicate that they really don't like it when people use it.

This at least provides some way for us to disable RTSS without crashing. The other problem is now trying to figure out when we should disable it, since the RTSS module itself doesn't provide any useful metadata to determine the version. Maybe we can scan the directory it's contained in for RTSS.exe, which does seem to include some version information.

[jellysquid3]

Resolved with 8878f17 and 4da6b37.