External App-Driven Execution Model Clean up Issues

[nibanks]

Describe the bug

With the latest external execution APIs, cleanup has issues. If any objects still have references internally then MsQuic waits for them to be cleaned up in both RegistrationClose and MsQuicClose. For external execution, this creates a deadlock because there might not be a different thread to run and drain any clean up work queued during these calls.

For instance, RegistrationClose may trigger async shutdown of any leftover connections, and then wait. MsQuicClose cleans up the stateless registration and datapath, both of which might queue other cleanup work; and then it waits.

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

Run a single threaded handshake test (server and client local) and then try to clean up.

Expected behavior

Cleanup should succeed.

Actual outcome

Depending on the situation, one of the MsQuic APIs hangs indefinitely.

Additional details

No response

[nibanks]

I've been thinking about this problem, and I think the solution is that we must expose async clean up (with completion callbacks) APIs for registrations and the API overall.

My idea: RegistrationCloseAsync and MsQuicCloseAsync with take a completion function pointer (and context pointer). These APIs effectively do the same thing as the synchronous APIs, but instead of waiting (usually on rundowns) they will return; then when the rest completes the callback functions will be called (very last thing, even after everything is freed).