NetworkChange on Linux: permanent deadlock between CloseSocket() under s_gate and the netlink reader's ProcessEvent callback

[alexeyzimarev]

Description

On Linux, removing the last NetworkAddressChanged / NetworkAvailabilityChanged subscriber can deadlock permanently against the netlink reader loop. Two threads form a cycle:

Unsubscriber — the remove accessor takes lock (s_gate) and, finding both subscriber collections empty, calls CloseSocket() → Socket.Dispose(). SafeSocketHandle.CloseAsIs then spins in while (!_released) { canceledOperations |= TryUnblockSocket(abortive); sw.SpinOnce(); } waiting for the last SafeHandle reference to be released. It holds s_gate for the entire spin.

Netlink reader — ReadEventsAsync is inside Interop.Sys.ReadEvents(socket.SafeHandle, &ProcessEvent), so it holds that reference, and the ProcessEvent callback's first act is lock (s_gate). The source comments on exactly this reference:

It's safe to compare raw handle values because ProcessEvents gets called from ReadEvents which holds a reference on the SafeHandle.

Neither side can progress, ever. TryUnblockSocket cannot break it — the reader is not blocked in a syscall, it is parked on a managed monitor.

Relevant code (permalinks pinned to 8ffe515):

• CloseSocket() call under lock (s_gate), NetworkAddressChanged remove accessor
• CloseSocket() call under lock (s_gate), NetworkAvailabilityChanged remove accessor
• CloseSocket() → Socket.Dispose()
• ProcessEvent taking lock (s_gate) from inside ReadEvents
• SafeSocketHandle.CloseAsIs spin

The current Socket-based design dates from #64614.

Why this is worse than a single hung call

The unsubscribe frequently runs on the finalizer thread, via HttpConnectionPoolManager+NetworkChangeCleanup.Finalize(). When it deadlocks there, the finalizer thread is gone for the life of the process:

• every GC.WaitForPendingFinalizers() anywhere in the process blocks forever;
• no finalizer ever runs again, so SafeHandles, sockets and file handles stop being reclaimed.

In our case 13 of 22 threads ended up parked in GC.WaitForPendingFinalizers() behind that one finalizer, and the process stopped doing useful work at all. It presented as an unexplained stall, not as an error.

Actual behaviour

From dotnet-dump analyze --command clrthreads on a heap dump of the stalled process:

Thread (finalizer):
  System.Private.CoreLib!System.Threading.Thread.Sleep(int32)
  System.Private.CoreLib!System.Threading.SpinWait.SpinOnceCore(int32)
  System.Net.Sockets!System.Net.Sockets.Socket.Dispose(bool)
  System.Net.Sockets!System.Net.Sockets.Socket.Dispose()
  System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.CloseSocket()
  System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.remove_NetworkAddressChanged(...)
  System.Net.Http!System.Net.Http.HttpConnectionPoolManager+NetworkChangeCleanup.Finalize()
  System.Private.CoreLib!System.GC.RunFinalizers()

Thread (threadpool):
  System.Private.CoreLib!System.Threading.Monitor.Enter_Slowpath(class System.Object)
  System.Private.CoreLib!System.Threading.Monitor.Enter(class System.Object,bool&)
  System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.ProcessEvent(int,value class NetworkChangeKind)
  System.Net.NetworkInformation!Interop+Sys.ReadEvents(class System.Runtime.InteropServices.SafeHandle,fnptr void(int,value class NetworkChangeKind))
  System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange+<ReadEventsAsync>d__29.MoveNext()
  ...

Plus 13 further threads blocked in System.GC.WaitForPendingFinalizers().

Expected behaviour

Unsubscribing the last handler completes, whether or not a netlink event is in flight.

Reproduction steps

No minimal repro — it is a race, hit in CI rather than constructed. The two conditions that make it reachable:

1. Repeatedly drive the subscriber count to zero, since each 1→0 transition is another CloseSocket(). Creating and releasing many SocketsHttpHandlers does this implicitly, because each pool manager subscribes and unsubscribes from its finalizer.
2. Generate address-change events continuously, so the reader is usually mid-dispatch — e.g. ip link add dummy0 type dummy plus ip addr add / ip addr del in a loop, or heavy container churn (every veth pair is an event).

Our occurrence was an integration test suite: several hundred host boots (so several hundred SocketsHttpHandlers created and finalized) while 12 Docker containers were being created and destroyed concurrently.

Workaround

Hold one NetworkAddressChanged subscription for the lifetime of the process. Both CloseSocket() call sites are guarded by "both subscriber collections are now empty", so a subscriber that is never removed makes them unreachable and the window never opens.

Possible direction for a fix

Offered tentatively — this is a reading of the code, not something I have built or tested.

Avoid disposing while holding s_gate: capture the socket and null the field under the lock, then dispose outside it. ProcessEvent already re-checks Socket != null && socket == Socket.Handle under the lock and would no-op, and ReadEventsAsync already catches ObjectDisposedException and SocketError.OperationAborted.

Configuration

• .NET 10.0.10, linux-x64
• Ubuntu 24.04.1 LTS, x64 VM

Regression?

Not known to be. The shape appears long-standing and is present on main.

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.