Please help me understand hookDetour

[quetzalsly]

I havent worked with hooks for a while, I remember doing it with microsoft detours library.
It was super simple because you just called it like hook(theirfunc, myfunc);

I am having trouble understanding what is nextInstructionOffset. Where am I supposed to be finding this information from? Inside IDA inside the actual function or at the place where the function is called?

I dont want to accidently overwrite any code and what if the function changes later? then I would have to keep rechecking these bytes in IDA every time?

I looked at the usage of this lib in GW2 as reference, I am having trouble understanding why hook(hl::CpuContext *ctx) is better than a standard hook like detours lib?

Also this:
void __fastcall hkGameThread(uintptr_t pInst, int, int frame_time)
{
    auto pCore = g_initObj.getMain();
    static auto orgFunc = ((void(__thiscall*)(uintptr_t, int))pCore->m_hkAlertCtx->getLocation());

    orgFunc(pInst, frame_time);
}

Why is the params have an extra int and the orgfunc does not? I remember doing something similar for a different calling declaration but not sure if it was __fastcall or __thiscall, sorry been so long since I messed around with it.

Would really appreciate an explanation, love the lib btw, the pattern scanning stuff is super good. It would be cool to be able to search for a function that has two string refs within it, because sometimes the function list in ida returns too many references and it would be cool to narrow it down to a single function easily.

[rafzi]

Thanks for your feedback!

The core issue with a patching jump hook like that is that you do not know where to jump back to. For example if the to-be-hooked function is (x86):

xor eax,eax  // 2 bytes
xor ecx,ecx  // 2 bytes
xor edx,edx  // 2 bytes
ret  // 1 byte

and it is overwritten with a 5 bytes jump, the jump back to the original code needs to be at offset 6 because an offset of 5 would jump in the middle of the third instruction. This can be automated with a disassembler and I believe this is what MS Detour does. You could just add Capstone to your project and let it disassemble the start of your target function to figure out where the first instruction after the hooking jump starts. I did not want to add such a larger dependency to hacklib.

If you do not need to jump back to the original function, you can also use hookJMP with an arbitrary offset.

Regarding the GW2 example, I do not remember that particular case, but it looks like a trick to get the ECX register content without actually defining a class member function. The fastcall calling convention uses the ECX and EDX registers to pass the first two arguments. Then we discard the second argument because EDX does not have any meaningful data. The original function seems to be a class member function, which can also be expressed as thiscall where the first argument is passed in ECX.

void __fastcall hkFunc(ECX, EDX (garbage), Stack...) {
  __thiscall orgFunc(ECX, Stack...)
}

[quetzalsly]

Thanks a lot of the explanation. I actually ended up using detours cos its free and open source now, works perfectly without having to count bytes manually.

I just had one more question about __thiscall vs __fastcall. Does IDA know which is which? I only ever see __fastcall in IDA and I hooked it exactly the same way as IDA shows and it seems to work but the game function is returning __int64 but in some code I saw on bitbucket they use void as the return type. Just wondering if this is some other quirk of IDA or its actually meant to return __int64?

[rafzi]

IDA can only guess what was originally defined in the source code and compiler. Many parts of calling conventions with argument and return signature are compatible, so it can be ambiguous which one is the "true" function signature.

For example, to figure out whether a function has a return value, I'd look at the callers and check if they use it.

[quetzalsly]

Thanks a lot, I will continue learning and using your library for many years :)