Skip to content

Fix macOS crash in NetworkChange when the run loop thread already exited - #131867

Open
isaacisland wants to merge 1 commit into
dotnet:mainfrom
isaacisland:fix-osx-networkchange-null-runloop
Open

Fix macOS crash in NetworkChange when the run loop thread already exited#131867
isaacisland wants to merge 1 commit into
dotnet:mainfrom
isaacisland:fix-osx-networkchange-null-runloop

Conversation

@isaacisland

Copy link
Copy Markdown

Problem

On macOS, NetworkChange.StopRunLoop() guards s_runLoop with only a Debug.Assert — erased in Release — and then dereferences it:

private static void StopRunLoop()
{
    Debug.Assert(s_runLoop != IntPtr.Zero);
    ...
    SpinWait.SpinUntil(() => Interop.RunLoop.CFRunLoopIsWaiting(s_runLoop));

RunLoopThreadStart() sets s_runLoop = IntPtr.Zero and returns whenever CFRunLoopRun() returns early, which happens once the SCDynamicStore run loop source is invalidated — a configd restart, or sleep/wake on a laptop. That path is silent: no exception is raised, subscribers stay registered, and nothing observes that the listener is gone (address-changed events simply stop firing from then on).

If the last subscriber is removed after that, StopRunLoop() passes NULL to CFRunLoopIsWaiting. CoreFoundation reads the CFRuntimeBase._cfinfo header at [NULL+8] and the process dies with EXC_BAD_ACCESS / KERN_INVALID_ADDRESS at 0x8. It's a SIGSEGV, so it cannot be caught — the whole process goes down.

Evidence

This is not theoretical. We hit it as the single largest crash in a fleet of macOS agents: 13,497 crashes across 6,213 machines over ~5 weeks. Our service is a long-lived daemon that subscribes to NetworkAddressChanged and unsubscribes during shutdown, so it reaches the 1→0 subscriber transition on every restart.

Across 25 crash dumps from distinct machines, the register context is identical:

  • x0 = 0x0 — the CFRunLoopRef argument
  • x8 = 0x8 — the computed fault address
  • lr inside CFRunLoopIsWaiting, pc in __CFCheckCFInfoPACSignature

and in all 25, the .NET Network Address Change thread is absent from the process — i.e. it had already run its epilogue and zeroed s_runLoop. Reproduced on macOS 15.1, 26.2, 26.3, 26.4, 26.5 and 26.6, so it is not tied to an OS version. StopRunLoop() is byte-identical from release/6.0 through release/10.0 and main, unchanged since #41768 (Oct 2019).

Most applications never hit this because they never unsubscribe — they just exit, and StopRunLoop() is never called.

Fix

Return early when s_runLoop is already zero.

Two details beyond a bare null check:

  • The early return still waits on s_runLoopEndedEvent. That thread has already called Set() (or is about to), and since it's an AutoResetEvent, leaving the signal unconsumed would make the next StopRunLoop() return immediately and desynchronize the start/stop state machine. Waiting also correctly handles the window where s_runLoop is zeroed but the epilogue hasn't finished.
  • The two remaining asserts move below the check — in this state the thread's epilogue has already nulled s_runLoopSource and s_dynamicStoreRef, so they would fire spuriously in Debug.

Notes / possible follow-ups

Happy to fold either of these in, or leave them as separate issues:

  1. The silent listener death is itself a bug. Once CFRunLoopRun() returns early, NetworkAddressChanged never fires again for the lifetime of the process, with no error surfaced. Recovering (re-creating the store and thread on the next subscribe) would be a behavior change, so I've left it out of this fix.
  2. SCDynamicStoreCreateRunLoopSource's result is never validity-checked in CreateAndStartRunLoop(), unlike the CFString/pattern creations just above it. A null source there would make CFRunLoopAddSource a no-op and CFRunLoopRun() return immediately — reaching the same broken state from the very first subscribe.

A deterministic regression test needs the listener thread forced through its epilogue (e.g. calling CFRunLoopStop on s_runLoop directly and waiting for it to zero) before removing the last handler. I didn't want to add a test that reaches into private static state without a maintainer's preference on approach — glad to add one in whatever form you'd like.

On macOS StopRunLoop() guards s_runLoop with only a Debug.Assert, which is
erased in Release, and then dereferences it via
CFRunLoopIsWaiting(s_runLoop).

RunLoopThreadStart() sets s_runLoop = IntPtr.Zero and returns whenever
CFRunLoopRun() returns early, which happens once the SCDynamicStore run loop
source is invalidated (for example a configd restart, or sleep/wake). That path
is silent: no exception is raised and subscribers stay registered, so nothing
observes that the listener is gone.

If the last subscriber is removed after that, StopRunLoop() passes NULL to
CFRunLoopIsWaiting. CoreFoundation reads the CFRuntimeBase _cfinfo header at
[NULL+8], and the process dies with EXC_BAD_ACCESS / KERN_INVALID_ADDRESS at
0x8. It is a SIGSEGV, so it cannot be caught.

Return early when s_runLoop is already zero. The early return still waits on
s_runLoopEndedEvent so the stale signal from the exited thread is consumed and a
later CreateAndStartRunLoop/StopRunLoop pair starts from a clean state, and the
two remaining asserts move below the check because that thread's epilogue has
already nulled s_runLoopSource and s_dynamicStoreRef.
@dotnet-policy-service dotnet-policy-service Bot added the community-contribution Indicates that the PR has been added by a community member label Aug 5, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-System.Net community-contribution Indicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant