TEST instruction using RIP is not updated in trampoline

[bo3b]

I'm getting a crash while hooking the nvapi function. It looks to me like In-Proc is not updating the relative addressing for the 'test' opcode using RIP register.

I'm using the latest version of the In-Proc code, cloned and built successfully.

---

Here is the original nvapi_QueryInterface function from nvapi64.dll.

00007FFD6C855E00  sub         rsp,28h  
00007FFD6C855E04  test        byte ptr [7FFD6CC52488h],4  
00007FFD6C855E0B  mov         r9d,ecx  
00007FFD6C855E0E  jne         00007FFD6C855E20  
00007FFD6C855E10  cmp         ecx,33C7358Ch  
...

After hooking, we get the debug output from In-Proc:

NktHookLib: Disassembly 0x7FFD6C855E00 -> sub rsp, 0x28
NktHookLib: Disassembly 0x7FFD6C855E04 -> test byte [rip+0x3fc67d], 0x4
NktHookLib: Hook installed. Proc @ 0x7FFD6C855E00 -> 0x7FFD53D19C50 (Stub @ 0x7FFD2C850000) 

The test byte ptr [7FFD6CC52488h],4 is using relative addressing off the RIP register. This is shown in In-Proc disassembly, although the syntax is slightly different in the VS dissassembly.

The function address then looks like (disassembly slightly off after jmp):

00007FFD6C855E00  jmp         00007FFD2C850000  
00007FFD6C855E05  add         eax,3FC67Dh  
00007FFD6C855E0A  add         al,44h  
00007FFD6C855E0C  mov         ecx,ecx  
00007FFD6C855E0E  jne         00007FFD6C855E20  
00007FFD6C855E10  cmp         ecx,33C7358Ch  
00007FFD6C855E16  je          00007FFD6C855E7E  

The trampoline function:

00007FFD2C850000  nop  
00007FFD2C850001  nop  
00007FFD2C850002  nop  
00007FFD2C850003  nop  
00007FFD2C850004  nop  
00007FFD2C850005  nop  
00007FFD2C850006  nop  
00007FFD2C850007  nop  
00007FFD2C850008  push        rdx  
00007FFD2C850009  mov         rdx,7FFD2C860000h  
00007FFD2C850013  test        qword ptr [rdx],101h  
00007FFD2C85001A  jne         00007FFD2C85002B  
00007FFD2C85001C  pop         rdx  
00007FFD2C85001D  jmp         qword ptr [7FFD2C850023h]  
00007FFD2C850023  push        rax  
00007FFD2C850024  pushfq  
00007FFD2C850025  rcl         dword ptr [rbx-3],1  
00007FFD2C850028  jg          00007FFD2C85002A  
00007FFD2C85002A  add         byte ptr [rax],al  
00007FFD2C85002C  sub         rsp,28h           <---- original function start
00007FFD2C850030  test        byte ptr [7FFD2CC4C6B4h],4    <----- Bad instruction ** crash
00007FFD2C850037  jmp         qword ptr [7FFD2C85003Dh]  
00007FFD2C85003D  or          ebx,dword ptr [rsi-7Bh]  
00007FFD2C850040  ins         byte ptr [rdi],dx  00007FFD2C850041  std  
00007FFD2C850042  jg          00007FFD2C850044  
00007FFD2C850044  add         byte ptr [rax],al  
00007FFD2C850046  add         byte ptr [rax],al  

The newly hooked destination function:

void* __cdecl Hooked_nvapi_QueryInterface(
	UINT32 offset)
{
00007FFD48E59BB0  mov         dword ptr [rsp+8],ecx  
00007FFD48E59BB4  push        rdi  
00007FFD48E59BB5  sub         rsp,30h  
00007FFD48E59BB9  mov         rdi,rsp  
00007FFD48E59BBC  mov         ecx,0Ch  
00007FFD48E59BC1  mov         eax,0CCCCCCCCh  
00007FFD48E59BC6  rep stos    dword ptr [rdi]  
00007FFD48E59BC8  mov         ecx,dword ptr [offset]  
00007FFD48E59BCC  lea         rcx,[__40B02ED0_inproc_dx11@cpp (07FFD48EA8071h)]  
00007FFD48E59BD3  call        __CheckForDebuggerJustMyCode (07FFD48E6C9C0h)  
	void* ptr = pOrignvapi_QueryInterface(offset);
00007FFD48E59BD8  mov         ecx,dword ptr [offset]  
00007FFD48E59BDC  call        qword ptr [pOrignvapi_QueryInterface (07FFD48E9E000h)]  
00007FFD48E59BE2  mov         qword ptr [ptr],rax  

	return ptr;
00007FFD48E59BE7  mov         rax,qword ptr [ptr]  
}
00007FFD48E59BEC  add         rsp,30h  
00007FFD48E59BF0  pop         rdi  
00007FFD48E59BF1  ret  

Minimized Cpp code implementing hook:

void* (__cdecl *pOrignvapi_QueryInterface)(
	UINT32 offset
	) = nullptr;

void* __cdecl Hooked_nvapi_QueryInterface(
	UINT32 offset)
{
	void* ptr = pOrignvapi_QueryInterface(offset);

	return ptr;
}

void HookNvapiQueryInterface()
{
	HMODULE hNvapi = LoadLibrary(L"nvapi64.dll");
	FARPROC pQueryInterface = GetProcAddress(hNvapi, "nvapi_QueryInterface");

	nktInProc.SetEnableDebugOutput(TRUE);

	SIZE_T hook_id;
	DWORD dwOsErr = nktInProc.Hook(&hook_id, (void**)&pOrignvapi_QueryInterface,
		pQueryInterface, Hooked_nvapi_QueryInterface, 0);
}

[mxmauro]

Hi @bo3b thanks for pointing out the issue.

Opcodes F6 05, F7 05 and 48 F7 05 were not handled at all but I'm adding the code.

Do you know some app/game using this NVidia dll to do tests?

Regards,
Mauro.

[bo3b]

@mxmauro Thanks for taking a look. Just FYI, no rush for me here, I've got a good workaround, just wanted to report it.

No known game/app using this on NVidia dll. I ran across it while prototyping a new tool.

If it's helpful for you, I can fairly easily make a test case from my Direct Mode sample program.

-bo3b

[mxmauro]

Hi @bo3b, don't worry. I'll create some dummy app to load the dll.

[mxmauro]

Hi again @bo3b . Uploaded the code (just source). Seems to work fine on my side.

[bo3b]

@mxmauro Looks good. Thanks very much for the timely response.

I'm having trouble verifying the result with my prototype, but could be something else.
So I went ahead and made a test branch in my DirectMode sample program to test this specific case. So in case it's helpful, here is a test case that demonstrates the crash, then the next commit fixes the crash using your latest code (as a prebuilt library).

https://github.com/bo3b/3D-Vision-Direct/tree/test_In-Proc

Looks good to me for the RIP test. I cannot speak to the other variants that you've fixed.

[bo3b]

@mxmauro Found out what the problem was with my test prototype- two different directories with NktHkLib, using the wrong one. After cleaning that up, I can confirm this works there as well.

Looks good, thanks very much.

[mxmauro]

Excellent!!