Add async cleanup APIs for external execution mode

docs/API.md

@@ -77,20 +77,21 @@ MsQuic supports a variety of configuration options available to both application
 
 ## Library Function Table
 
-There are only two top level functions:
+There are a few top level functions:
 
 - [MsQuicOpenVersion](api/MsQuicOpenVersion.md) - Initializes the MsQuic library and returns a the API function table.
 - [MsQuicClose](api/MsQuicClose.md) - Cleans up the function table and releases the library reference from the previous [MsQuicOpenVersion](api/MsQuicOpenVersion.md) call.
+- [CloseAsync](api/CloseAsync.md) - Asynchronously cleans up the function table and releases the library reference (Available from v2.6).
 
-When the app is done with the MsQuic library, it **must** call [MsQuicClose](api/MsQuicClose.md) and pass in the function table it received from [MsQuicOpenVersion](api/MsQuicOpenVersion.md). This allows for the library state to be cleaned up.
+When the app is done with the MsQuic library, it **must** call either [MsQuicClose](api/MsQuicClose.md) or [CloseAsync](api/CloseAsync.md) and pass in the function table it received from [MsQuicOpenVersion](api/MsQuicOpenVersion.md). This allows for the library state to be cleaned up.
 
 Please note, there is no explicit start/stop API for this library. Each API function table has a reference on the QUIC library: the library is initialized when the first call to [MsQuicOpenVersion](api/MsQuicOpenVersion.md) succeeds and uninitialized when the last call to [MsQuicClose](api/MsQuicClose.md) completes. An app should therefore beware of repeatedly calling [MsQuicOpenVersion](api/MsQuicOpenVersion.md) and [MsQuicClose](api/MsQuicClose.md), as library setup/cleanup can be expensive.
 
 ## Registration
 
 Generally, each app only needs a single registration. The registration represents the execution context where all logic for the app's connections run. The library will create a number of worker threads for each registration, shared for all the connections. This execution context is not shared between different registrations.
 
-A registration is created by calling [RegistrationOpen](api/RegistrationOpen.md) and deleted by calling [RegistrationClose](api/RegistrationClose.md).
+A registration is created by calling [RegistrationOpen](api/RegistrationOpen.md) and deleted by calling either [RegistrationClose](api/RegistrationClose.md) or [RegistrationCloseAsync](api/RegistrationCloseAsync.md) (Available from v2.6).
 
 ## Configuration
 

docs/PreviewFeatures.md

@@ -40,3 +40,12 @@ TODO
 [StreamProvideReceiveBuffers](StreamProvideReceiveBuffers.md)
 
 TODO
+
+### Asynchronous Close APIs
+
+MsQuic now provides asynchronous alternatives to the synchronous close functions to avoid deadlocks in single-threaded execution environments:
+
+- [RegistrationCloseAsync](api/RegistrationCloseAsync.md) - Asynchronously closes a registration handle
+- [CloseAsync](api/CloseAsync.md) - Asynchronously cleans up the function table and releases the library reference
+
+These APIs are particularly useful in scenarios where synchronous close operations might cause deadlocks, such as when closing objects from within MsQuic callbacks or in single-threaded execution environments.

docs/api/CloseAsync.md

@@ -0,0 +1,37 @@
+CloseAsync function
+======
+
+Asynchronously closes an existing handle to the MsQuic library, releasing the reference on the library and freeing the function table.
+
+# Syntax
+
+```C
+typedef
+_IRQL_requires_max_(PASSIVE_LEVEL)
+void
+(QUIC_API *QUIC_CLOSE_ASYNC_FN)(
+    _In_ QUIC_COMPLETE_HANDLER Handler,
+    _In_opt_ void* Context
+    );
+```
+
+# Parameters
+
+`Handler`
+
+The callback handler to be invoked when the close operation is complete.
+
+`Context`
+
+Optional context pointer to be passed to the Handler when the close operation completes.
+
+# Remarks
+
+This function is an asynchronous alternative to [MsQuicClose](MsQuicClose.md). Unlike [MsQuicClose](MsQuicClose.md), this function will not block and instead, registers a callback to be invoked when the cleanup is complete. This avoids deadlocks in single-threaded execution environments.
+
+This function (or [MsQuicClose](MsQuicClose.md)) **must** be called when the app is done with the MsQuic library.
+
+# See Also
+
+[MsQuicOpenVersion](MsQuicOpenVersion.md)<br>
+[MsQuicClose](MsQuicClose.md)<br>

docs/api/RegistrationCloseAsync.md

@@ -0,0 +1,55 @@
+RegistrationCloseAsync function
+======
+
+Asynchronously closes an existing registration.
+
+# Syntax
+
+```C
+typedef
+_IRQL_requires_max_(PASSIVE_LEVEL)
+QUIC_STATUS
+(QUIC_API * QUIC_REGISTRATION_CLOSE_ASYNC_FN)(
+    _In_ _Pre_defensive_ __drv_freesMem(Mem)
+        HQUIC Registration,
+    _In_ QUIC_COMPLETE_HANDLER Handler,
+    _In_opt_ void* Context
+    );
+```
+
+# Parameters
+
+`Registration`
+
+A registration handle from a previous call to [RegistrationOpen](RegistrationOpen.md).
+
+`Handler`
+
+The callback handler to be invoked when the registration close operation is complete.
+
+`Context`
+
+Optional context pointer to be passed to the Handler when the close operation completes.
+
+# Return Value
+
+The function returns a [QUIC_STATUS](QUIC_STATUS.md). The app may use `QUIC_FAILED` or `QUIC_SUCCEEDED` to determine if the function failed or succeeded.
+
+* **QUIC_STATUS_SUCCESS** - The operation completed synchronously. The callback handler will **not** be called.
+* **QUIC_STATUS_PENDING** - The operation will complete asynchronously. The callback handler will be called upon completion.
+* Any other status code indicates an error.
+
+# Remarks
+
+Unlike [RegistrationClose](RegistrationClose.md), this function will not block waiting for child objects to be cleaned up. Instead, if the operation can't complete immediately, it registers a callback to be invoked when the cleanup is complete. This avoids deadlocks in single-threaded execution environments.
+
+If the function returns QUIC_STATUS_SUCCESS, the cleanup completed synchronously and the callback will not be invoked. If it returns QUIC_STATUS_PENDING, the callback will be invoked when the cleanup is complete.
+
+The application **must** close/delete all child configurations and connection objects before closing the registration. This call will trigger the provided callback when those outstanding objects are cleaned up, only if the function returns QUIC_STATUS_PENDING.
+
+It is safe to call this function from any MsQuic event callback, as it will not deadlock.
+
+# See Also
+
+[RegistrationOpen](RegistrationOpen.md)<br>
+[RegistrationClose](RegistrationClose.md)<br>

src/core/api.h

@@ -5,6 +5,21 @@
 
 --*/
 
+typedef struct QUIC_API_TABLE_EX {
+    QUIC_API_TABLE Table; // Must be first!
+    QUIC_COMPLETE_HANDLER CompleteHandler;
+    void* CompleteContext;
+} QUIC_API_TABLE_EX;
+
+_IRQL_requires_max_(PASSIVE_LEVEL)
+void
+QUIC_API
+MsQuicCloseAsync(
+    _In_ _Pre_defensive_ const void* QuicApi,
+    _In_ QUIC_COMPLETE_HANDLER Handler,
+    _In_opt_ void* Context
+    );
+
 _IRQL_requires_max_(PASSIVE_LEVEL)
 QUIC_STATUS
 QUIC_API
@@ -48,6 +63,16 @@ MsQuicRegistrationClose(
         HQUIC Handle
     );
 
+_IRQL_requires_max_(PASSIVE_LEVEL)
+QUIC_STATUS
+QUIC_API
+MsQuicRegistrationCloseAsync(
+    _In_ _Pre_defensive_ __drv_freesMem(Mem)
+        HQUIC Handle,
+    _In_ QUIC_COMPLETE_HANDLER Handler,
+    _In_opt_ void* Context
+    );
+
 _IRQL_requires_max_(DISPATCH_LEVEL)
 void
 QUIC_API

src/core/configuration.c

@@ -194,7 +194,7 @@ MsQuicConfigurationOpen(
 
     QuicConfigurationSettingsChanged(Configuration);
 
-    BOOLEAN Result = CxPlatRundownAcquire(&Registration->Rundown);
+    BOOLEAN Result = QuicRegistrationAddRef(Registration);
     CXPLAT_FRE_ASSERT(Result);
 
     CxPlatLockAcquire(&Registration->ConfigLock);
@@ -255,7 +255,7 @@ QuicConfigurationUninitialize(
 
     QuicSettingsCleanup(&Configuration->Settings);
 
-    CxPlatRundownRelease(&Configuration->Registration->Rundown);
+    QuicRegistrationRelease(Configuration->Registration);
 
     QuicTraceEvent(
         ConfigurationDestroyed,

src/core/connection.c

@@ -392,7 +392,7 @@ QuicConnFree(
         QuicPerfCounterDecrement(Partition, QUIC_PERF_COUNTER_CONN_CONNECTED);
     }
     if (Connection->Registration != NULL) {
-        CxPlatRundownRelease(&Connection->Registration->Rundown);
+        QuicRegistrationRelease(Connection->Registration);
     }
     if (Connection->CloseReasonPhrase != NULL) {
         CXPLAT_FREE(Connection->CloseReasonPhrase, QUIC_POOL_CLOSE_REASON);
@@ -473,7 +473,7 @@ QuicConnUnregister(
         CxPlatDispatchLockAcquire(&Connection->Registration->ConnectionLock);
         CxPlatListEntryRemove(&Connection->RegistrationLink);
         CxPlatDispatchLockRelease(&Connection->Registration->ConnectionLock);
-        CxPlatRundownRelease(&Connection->Registration->Rundown);
+        QuicRegistrationRelease(Connection->Registration);
 
         QuicTraceEvent(
             ConnUnregistered,
@@ -495,7 +495,7 @@ QuicConnRegister(
 {
     QuicConnUnregister(Connection);
 
-    if (!CxPlatRundownAcquire(&Registration->Rundown)) {
+    if (!QuicRegistrationAddRef(Registration)) {
         return FALSE;
     }
     Connection->State.Registered = TRUE;
@@ -518,7 +518,7 @@ QuicConnRegister(
     if (RegistrationShuttingDown) {
         Connection->State.Registered = FALSE;
         Connection->Registration = NULL;
-        CxPlatRundownRelease(&Registration->Rundown);
+        QuicRegistrationRelease(Registration);
     } else {
         QuicTraceEvent(
             ConnRegistered,