Updates for VR support in Madness engine games such as AMS2

[Jabbah]

PR to open discussion for AMS2 VR.

These are a few changes to enable ReShade to work on Automobilista 2 in VR. The current implementation and the existing PRs from fholger don't work for AMS2 so I did a little digging. AMS2 / PC2 (Madness Engine) might be a bit of an oddity as you can only load it through Steam for VR mode and the dxgi.dll and d3d11.dll libraries are supplied through SteamVR (or at least are loaded from the Steam folder). This means that the correct name for injecting the ReShade dll doesn't work. However using opengl32.dll does allow it to be loaded, even though it's not an opengl game.

Additionally it uses some old OpenVR helper functions to initialise and submit that aren't caught by the current hooks. I've added new hooks to allow ReShade to load and apply shaders to VR successfully, however because the GetGenericInterface function doesn't get caught by the existing hook we can't tell what version of the compositor is used, so the installed hook for the submit is specific to AMS2 / PC2 and may not work for other games that use the old VRInit method. I tried iterating through all possible interface versions and manually calling GetGenericInterface but they all worked up to the current 027 version. Not sure how else this could be done?

Another interesting issue was that the call to CreateTexture2D failed in the d3d11 runtime on submit. It appears that AMS2 also requires D3D11_BIND_SHADER_RESOURCE flag to be set. I tested this on a couple of other games and it still works but this probably needs more thought.

[crosire]

I went through the OpenVR history and this is where these changed:

• v0.9.17: Replaces VR_Init() export with VR_InitInternal()
  Also replaces VRCompositor() export with inline function that uses VR_GetGenericInterface
• v1.0.10: Replaces VR_InitInternal() with VR_InitInternal2()

So looks like AMS2 is using an OpenVR version from before v0.9.17. That versioned bumped the compositor interface version to IVRCompositor_012, so that roughly matches up with it using IVRCompositor_010, which means it has to be using version v0.9.14 (https://github.com/ValveSoftware/openvr/blob/v0.9.14/headers/openvr.h).

But based on those headers the VR_Init signature should be

HOOK_EXPORT vr::IVRSystem *VR_CALLTYPE VR_Init(vr::EVRInitError *peError, vr::EVRApplicationType eApplicationType);

instead of

HOOK_EXPORT uint32_t VR_CALLTYPE VR_Init(vr::EVRInitError* peError, vr::EVRApplicationType eApplicationType, const char* pStartupInfo = nullptr);

[crosire]

There should be no need for a set here. Can do something like this instead:

HOOK_EXPORT vr::IVRCompositor *VR_CALLTYPE VRCompositor()
{
	static vr::IVRCompositor *last_compositor_instance = nullptr;
	vr::IVRCompositor *const compositor_instance = reshade::hooks::call(VRCompositor)();
	if (compositor_instance != last_compositor_instance)
	{
		// This is just for ProjectCARS2 / AMS2 and done through trial and error (IVRCompositor_010)
		// Need a way to work out what the Compositor version is. VR_GetGenericInterface isn't called so don't get to hook it there.
		reshade::hooks::install("IVRCompositor::Submit", vtable_from_instance(static_cast<vr::IVRCompositor*>(compositor_instance)), 4, reinterpret_cast<reshade::hook::address>(IVRCompositor_Submit_009));
		last_compositor_instance = compositor_instance;
	}
	return compositor_instance;
}

[Jabbah]

Agreed, I should have updated this as it was to check if there were multiple calls.

In the absence of a way to get the interface versions such as with PC2 / AMS2, would it be worth adding an optional section to an ini file that allows OpenVR interface versions to be specified and then the hook function choice overriden if that section exists?

[crosire]

I dug through the DLL a bit, and it appears VRCompositor is calling IVRClientCore::GetGenericInterface behind the scenes. The same function VR_GetGenericInterface also calls (https://github.com/ValveSoftware/openvr/blob/4c85abcb7f7f1f02adaf3812018c99fc593bc341/src/openvr_api_public.cpp#L185). So I think only thing we need to do is instead of hooking, VR_GetGenericInterface, hook IVRClientCore::GetGenericInterface and all problems would be solved.
Only issue is, how to hook the IVRClientCore interface ... Only way to do that I can see is to hook VRClientCoreFactory function from vrclient.dll/vrclient_x64.dll (which is loaded by openvr_api.dll on init). Will need to try that.

Anyway, for now this is a temporary solution that is fine. In future the VR_Init(...) + VR_GetGenericInterface hooks can be replaced with VRClientCoreFactory + IVRClientCore::GetGenericInterface hooks that should hopefully solve all versioning issues. Just need to find a game to test this with.

[crosire]

@Jabbah Would be awesome if you could verify whether 4a58ad1 still works with AMS2!

[Jabbah]

@Jabbah Would be awesome if you could verify whether 4a58ad1 still works with AMS2!

Yes, fresh checkout of main still works with AMS2. You can't turn ReShade on and off once it's running, something that fholgers changes fix, but that is the same behaviour as before so changes look good.