ExecutionEngineException after GC.Collect when using global hook

[Germs2004]

Issue Title

Reproducible crash caused by garbage collection after installing a global keyboard hook.

General

The crash is reproducible every time. I created a whole new project to demo the problem. Here's how it works:

1. Run the .Net Core 3.1 desktop app.
2. It adds a global keyboard hook using Windows API
3. The hook works fine and you can press lots of keys, no problem.
4. Garbage collection runs
5. Now, the first key you press, it will crash the program with ExecutionEngineException. It seems that garbage collection breaks something with the keyboard hook.

I can't find anybody else with this problem and can't find any solution. Several people have reported a different crash with ExecutionEngineException where disabling ConcurrentGarbageCollection solves their issue, but that solution does not work with my problem. Am I doing something wrong, or is there some workaround I can use until this gets fixed upstream?

Windows 10 2004, all updates applied
.Net Core 3.1
Visual Studio 2019 16.7.5

Very simple Visual Studio solution attached as a Zip file to demonstrate the issue.

HookTest.zip

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[Germs2004]

After a week of searching, I finally found a workaround for this problem:

Instead of passing the method name directly with your call to SetWindowsHookEx, create a static variable to hold that method's reference, and pass that static variable into SetWindowsHookEx. That way, garbage collection won't "clean up" the reference that the hook API needs to keep around.

Bad code that breaks after garbage collection:

private static IntPtr Process_Keyboard_Keys(int nCode, IntPtr wParam, IntPtr lParam)
{
// your method that receives the callback from the Windows hook
}

...
hookHandle = SetWindowsHookEx(WH_KEYBOARD_LL, Process_Keyboard_Keys, moduleHandle, 0U);

Good code that works after garbage collection:

private delegate IntPtr MyHOOKPROC(int code, IntPtr wParam, IntPtr lParam);
private static MyHOOKPROC myHookProc = new MyHOOKPROC(Process_Keyboard_Keys);

private static IntPtr Process_Keyboard_Keys(int nCode, IntPtr wParam, IntPtr lParam)
{
// your method that receives the callback from the Windows hook
}

...
hookHandle = SetWindowsHookEx(WH_KEYBOARD_LL, myHookProc, moduleHandle, 0U);

source

[svick]

@Germs2004 Just a small note: that's not a workaround, that is the correct solution.

To invoke the delegate, the delegate has to still exist. Under most circumstances, the GC is smart enough to figure out when to keep the delegate around. But when you pass it to native function that keeps it around after the function returns, GC can't see that, so you have to help it by keeping another reference around. And your static field does exactly that.