Fix and extend Windows API hooking

[upside2]

Fixing bugs in API hooking of Windows functions GetModuleHandleExW() and _alloca_probe(), and adding more vamp signatures for _alloca_probe(). The correct stack adjustment in _alloca_probe() is important for deciding if variables are local, and in guessing how many arguments are passed to a function.

The signatures cover 32-bit and 64-bit executables from all Visual Studio versions; it turns out the opcodes change a little over time, and vamp was only covering some of the 32-bit cases. One of the existing signatures was misnamed.

The GetModuleHandleExW() hook had a typo.

The _alloca_probe() hook had a few problems: stack size delta is off by 4 bytes, stack size not safeguarded against going negative (which can happen during emulation if EAX is garbage), and needless execution of the page read loop which in real samples can take up to 1.5 secs of CPU time.

I created a unit test that leverages existing VIVTESTFILE samples.

[mr-tz]

This may be a fix for #515.

[atlas0fd00m]

nice work, thank you!
good fixes, new VAMP sigs, easy to read, and unittests so we know when we break it!

i'll wait a day or two before merging in case @rakuy0 has any concerns.

thanks @upside2 !

@

[atlas0fd00m]

@mr-tz can you validate this is a fix for #515 ?