Production server freeze/deadlock on .NET 9.0.7, CPU 0, gdb points to __GI_cfsetispeed in multiple threads

[vrecluse]

Description

Environment

• .NET Runtime Version: 9.0.7
• Operating System: Linux Debian 12
• Architecture: x64
• GC Mode: Server GC
• Environment Variable: DOTNET_GCName=libclrgc.so

Problem Description

We are running a .NET 9 server application in our production environment. The server application intermittently experiences a total freeze or deadlock, becoming completely unresponsive.

Observed Symptoms

When the freeze occurs, we observe the following:

1. No Response: All HTTP API endpoints stop responding.
2. No Logs: The application stops writing any new logs.
3. Delayed Timers: We have System.Threading.Timer instances that are observed to trigger significantly later than their scheduled time, indicating the entire process is stalled.
4. Correlation with GC: The freeze often correlates with a high-activity GC period (e.g., GC pause times reported as > 2 seconds just before the incident).
5. Process State: During the freeze, htop or top shows the process's CPU usage drops to 0.
6. Thread Pool: The thread pool queue length drops to near 0, even as new requests should be queuing.
7. Diagnostics Fail: Attempting to use dotnet-stack report on the hung process also hangs indefinitely and provides no output.

GDB Stack Traces

We successfully attached gdb to the hung process. We found that the vast majority of .NET TP Worker threads are idle, but a few critical threads seem to be involved in a deadlock.

1. Suspicious Stacks (Potential Deadlock)

Two threads, including the crucial .NET SynchManag thread, are stuck in __GI_cfsetispeed. This appears to be the core of the deadlock.

Thread 2 (Thread 0x7fb6b2fb56c0 (LWP 7) ".NET SynchManag"):
#0  0x00007fb6b38c121f in __GI_cfsetispeed (termios_p=0x7fb6b2fb4d78, speed=1) at ../sysdeps/unix/sysv/linux/speed.c:96
#1  0x00007fb6b2fb4db0 in ?? ()
#2  0x00007fb6b2fb4d78 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0xffffffff00000000 in ?? ()
#5  0x000055a8504b0f58 in ?? ()
#6  0x00007fb6b3628fa0 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#7  0x00007fb6b3628603 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#8  0x00007fb6b363214e in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#9  0x00007fb6b384e1f5 in __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>, start_routine=<optimized out>, arg=<optimized out>) at ./nptl/pthread_create.c:846
#10 0x0000000000000000 in ?? ()

Thread 181 (Thread 0x7e83927fc6c0 (LWP 255) "rdk:broker10265"):
#0  0x00007fb6b38c121f in __GI_cfsetispeed (termios_p=0x7e83740049b8, speed=2) at ../sysdeps/unix/sysv/linux/speed.c:96
#1# 0x0000000000000000 in ?? ()

2. Idle Worker Threads

Most other threads, especially .NET TP Worker threads, are idle and appear to be waiting for work (stuck in __pthread_attr_extension).

Thread 194 (Thread 0x7e8651ffb6c0 (LWP 311668) ".NET TP Worker"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x0000000000000189 in ?? ()
#2  0x000055a8504e83b8 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 193 (Thread 0x7e84fdffb6c0 (LWP 311660) ".NET TP Worker"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x000055a800000189 in ?? ()
#2  0x000055a8504e83b8 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 192 (Thread 0x7e851effd6c0 (LWP 311650) ".NET TP Worker"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x0000000000000189 in ?? ()
#2  0x000055a8504e83b8 in ?? ()
#3  0x0000000000000000 in ?? ()

Summary

The presence of __GI_cfsetispeed in the stack for the .NET SynchManag thread is highly suspicious and suggests a potential native-level deadlock, possibly within glibc or how the runtime interacts with it. This stall prevents all managed code (including timers and the thread pool) from making progress.

Given that diagnostic tools like dotnet-stack also fail, this points to a very low-level lock or stall.

---

Reproduction Steps

Hard to reproduce, occured sometimes in production environment

Expected behavior

Should not hang forever

Actual behavior

Hangs forever

Regression?

No response

Known Workarounds

No response

Configuration

• .NET Runtime Version: 9.0.7
• Operating System: Linux (inferred from stack trace)
• Architecture: x64 (inferred from stack trace)
• GC Mode: Server GC
• Environment Variable: DOTNET_GCName=libclrgc.so

Other information

No response

[am11]

The presence of __GI_cfsetispeed in the stack for the .NET SynchManag thread is highly suspicious and suggests a potential native-level deadlock, possibly within glibc or how the runtime interacts with it. This stall prevents all managed code (including timers and the thread pool) from making progress.

Do you have full backtrace to __GI_cfsetispeed? You can use dotnet symbol to download the native runtime symbols:

$ dotnet tool install --global dotnet-symbol
$ dotnet symbol --recurse-subdirectories --timeout 20 --symbols '/usr/share/dotnet/*.so'

The only place we are (explicitly) calling cfsetispeed(3p) is in src/native/libs/System.IO.Ports.Native/pal_termios.c, so unless you are using System.IO.Ports to connect to serial/comm interfaces etc. that's likely something outside the runtime causing it.

[vrecluse]

The presence of __GI_cfsetispeed in the stack for the .NET SynchManag thread is highly suspicious and suggests a potential native-level deadlock, possibly within glibc or how the runtime interacts with it. This stall prevents all managed code (including timers and the thread pool) from making progress.

Do you have full backtrace to __GI_cfsetispeed? You can use dotnet symbol to download the native runtime symbols:

$ dotnet tool install --global dotnet-symbol
$ dotnet symbol --recurse-subdirectories --timeout 20 --symbols '/usr/share/dotnet/*.so'
The only place we are (explicitly) calling cfsetispeed(3p) is in src/native/libs/System.IO.Ports.Native/pal_termios.c, so unless you are using System.IO.Ports to connect to serial/comm interfaces etc. that's likely something outside the runtime causing it.

No, the server has been restarted. We don't use System.IO.Ports directly. Any other clues or workarounds? What can I do if next time it happens again?

[am11]

It would be nice if you could get the native symbols uploaded to the server then capture the error logs and/or coredump.

For selfhosted/singlefile publish, you could uploaded the .dbg file(s) by running dotnet-symbol command on build machine. For framework-dependent deployment (e.g. on VM server), you could run dotnet-symbol command on the target machine as shown above (which will place .dbg files for all native libs like libcoreclr.so etc. next to the shared libs).

[vrecluse]

It would be nice if you could get the native symbols uploaded to the server then capture the error logs and/or coredump.

For selfhosted/singlefile publish, you could uploaded the .dbg file(s) by running dotnet-symbol command on build machine. For framework-dependent deployment (e.g. on VM server), you could run dotnet-symbol command on the target machine as shown above (which will place .dbg files for all native libs like libcoreclr.so etc. next to the shared libs).

I captured core dump with gcore.
Does this commands require excactly the same environment as the core dump generated (os version, dotnet runtime version, etc)?

$ dotnet tool install --global dotnet-symbol
$ dotnet symbol --recurse-subdirectories --timeout 20 --symbols '/usr/share/dotnet/*.so'

[am11]

It depends on how you published the app. If you are using:

• framework-dependent deployment (i.e. you have dotnet runtime installed on the server), then you'd need to run dotnet symbol --recurse.. command on the server by adjusting the path to dotnet install directory.
• for self-contained or singlefile apps, you can run the symbol command for your publish directory which will download all the required .dbg files next to the binaries on build machine before deployment: find /path/to/your-publish-dir -type f -exec sh -c 'for f; do file -L --mime-type "$f" 2>/dev/null | grep -qE "application/x-(sharedlib|(pie-)?executable)" && echo "$f"; done' _ {} + | xargs dotnet symbol --timeout 20

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[vrecluse]

It depends on how you published the app. If you are using:

• framework-dependent deployment (i.e. you have dotnet runtime installed on the server), then you'd need to run dotnet symbol --recurse.. command on the server by adjusting the path to dotnet install directory.
• for self-contained or singlefile apps, you can run the symbol command for your publish directory which will download all the required .dbg files next to the binaries on build machine before deployment: find /path/to/your-publish-dir -type f -exec sh -c 'for f; do file -L --mime-type "$f" 2>/dev/null | grep -qE "application/x-(sharedlib|(pie-)?executable)" && echo "$f"; done' _ {} + | xargs dotnet symbol --timeout 20

It's framework-dependent deployment on k8s. The noted stacks with symbols:

Thread 2 (Thread 0x7fb6b2fb56c0 (LWP 7) ".NET SynchManag"):
#0  0x00007fb6b38c121f in __GI_cfsetispeed (termios_p=0x7fb6b2fb4d78, speed=1) at ../sysdeps/unix/sysv/linux/speed.c:96
#1  0x00007fb6b2fb4db0 in ?? ()
#2  0x00007fb6b2fb4d78 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0xffffffff00000000 in ?? ()
#5  0x000055a8504b0f58 in ?? ()
#6  0x00007fb6b3628fa0 in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe (this=this@entry=0x55a85050bbc0, iTimeout=iTimeout@entry=-1, pRecvBuf=pRecvBuf@entry=0x7fb6b2fb4df7 "", iBytes=iBytes@entry=1) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:2238
#7  0x00007fb6b3628603 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (this=0x55a85050bbc0, iPollTimeout=-1, pswcWorkerCmd=<optimized out>, pshridMarshaledData=<optimized out>, pdwData=<optimized out>) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:2016
#8  CorUnix::CPalSynchronizationManager::WorkerThread (pArg=0x55a85050bbc0) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:1719
#9  0x00007fb6b363214e in CorUnix::CPalThread::ThreadEntry (pvParam=0x55a8504b0f40) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1744
#10 0x00007fb6b384e1f5 in __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>, start_routine=<optimized out>, arg=<optimized out>) at ./nptl/pthread_create.c:846
#11 0x0000000000000000 in ?? ()


Thread 67 (Thread 0x7e8764be66c0 (LWP 73) ".NET SigHandler"):
#0  0x00007fb6b38bd29c in __getcwd_generic (size=<optimized out>, buf=<optimized out>) at ../sysdeps/posix/getcwd.c:464
#1  __GI___getcwd (buf=<optimized out>, size=<optimized out>) at ../sysdeps/unix/sysv/linux/getcwd.c:110
#2  0x0000000000000000 in ?? ()

Thread 108 (Thread 0x7e85fbfff6c0 (LWP 168) ".NET File Watch"):
#0  0x00007fb6b38bd29c in __getcwd_generic (size=<optimized out>, buf=<optimized out>) at ../sysdeps/posix/getcwd.c:464
#1  __GI___getcwd (buf=<optimized out>, size=<optimized out>) at ../sysdeps/unix/sysv/linux/getcwd.c:110
#2  0x00007fb6b36cfb38 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#3  0x00000000fffffffc in ?? ()
#4  0x00007e85f4001cc8 in ?? ()
#5  0x00007eb71c72e3e0 in ?? ()
#6  0x00007eb71c303450 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 122 (Thread 0x7e84ab7fe6c0 (LWP 182) ".NET BGC"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x00007fb600000189 in ?? ()
#2  0x00007fb6b3718404 in CorUnix::CPalSynchronizationManager::s_csSynchProcessLock () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#3  0x0000000000000000 in ?? ()

Thread 126 (Thread 0x7e84a97fa6c0 (LWP 186) ".NET BGC"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0xffffffff00000189 in ?? ()
#2  0x00007fb6b3718404 in CorUnix::CPalSynchronizationManager::s_csSynchProcessLock () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#3  0x0000000000000000 in ?? ()

Thread 151 (Thread 0x7e842affd6c0 (LWP 211) ".NET BGC"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x0000000000000189 in ?? ()
#2  0x00007fb6b3718404 in CorUnix::CPalSynchronizationManager::s_csSynchProcessLock () from /usr/share/dotnet/shared/Microsoft.NETCore.App/9.0.7/libcoreclr.so
#3  0x0000000000000000 in ?? ()


Thread 179 (Thread 0x7e83c0ff96c0 (LWP 251) "rdk:broker10265"):
#0  0x00007fb6b38c121f in __GI_cfsetispeed (termios_p=0x7e8384005a98, speed=2) at ../sysdeps/unix/sysv/linux/speed.c:96
#1  0x0000000000000000 in ?? ()

The other call stacks are still like

Thread 220 (Thread 0x7e85d9ffb6c0 (LWP 311745) ".NET TP Worker"):
#0  0x00007fb6b384af16 in __pthread_attr_extension (attr=0x0) at ./nptl/pthread_attr_extension.c:30
#1  0x0000000000000189 in ?? ()
#2  0x000055a8504e83b8 in ?? ()
#3  0x0000000000000000 in ?? ()

@am11

[vrecluse]

related locals:

(gdb) bt

#0  0x00007fb6b38c121f in __GI_cfsetispeed (termios_p=0x7fb6b2fb4d78, speed=1) at ../sysdeps/unix/sysv/linux/speed.c:96
#1  0x00007fb6b2fb4db0 in ?? ()
#2  0x00007fb6b2fb4d78 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0xffffffff00000000 in ?? ()
#5  0x000055a8504b0f58 in ?? ()
#6  0x00007fb6b3628fa0 in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe (this=this@entry=0x55a85050bbc0, iTimeout=iTimeout@entry=-1, pRecvBuf=pRecvBuf@entry=0x7fb6b2fb4df7 "", iBytes=iBytes@entry=1)
    at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:2238
#7  0x00007fb6b3628603 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (this=0x55a85050bbc0, iPollTimeout=-1, pswcWorkerCmd=<optimized out>, pshridMarshaledData=<optimized out>, pdwData=<optimized out>)
    at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:2016
#8  CorUnix::CPalSynchronizationManager::WorkerThread (pArg=0x55a85050bbc0) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:1719
#9  0x00007fb6b363214e in CorUnix::CPalThread::ThreadEntry (pvParam=0x55a8504b0f40) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1744
#10 0x00007fb6b384e1f5 in __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>, start_routine=<optimized out>, arg=<optimized out>) at ./nptl/pthread_create.c:846
#11 0x0000000000000000 in ?? ()
(gdb) info locals

iErrno = 0
Poll = {fd = 3, events = 1, revents = 0}
iRet = <optimized out>
iConsecutiveEintrs = <optimized out>
iBytesRead = 0
pPos = 0x7fb6b2fb4df7 ""
(gdb) up

#7  0x00007fb6b3628603 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (this=0x55a85050bbc0, iPollTimeout=-1, pswcWorkerCmd=<optimized out>, pshridMarshaledData=<optimized out>, pdwData=<optimized out>)
    at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:2016
2016    in /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp
(gdb) info locals

swcWorkerCmd = CorUnix::CPalSynchronizationManager::SynchWorkerCmdNop
byVal = 0 '\000'
iRet = <optimized out>
shridMarshaledId = <optimized out>
dwData = <optimized out>
(gdb) up

#8  CorUnix::CPalSynchronizationManager::WorkerThread (pArg=0x55a85050bbc0) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:1719
1719    in /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp
(gdb) info locals

lProcessCount = <optimized out>
fShuttingDown = false
fWorkerIsDone = false
iPollTimeout = -1
pthrWorker = 0x55a8504b0f40
shridMarshaledData = <optimized out>
palErr = <optimized out>
swcCmd = <optimized out>
dwData = <optimized out>
twrWakeUpReason = <optimized out>
ptnwdWorkerThreadNativeData = <optimized out>
iRet = <optimized out>
pSynchManager = <optimized out>
(gdb) 

[am11]

.NET SynchManag thread might not be interesting, but BGC ones seem like runtime is stalling in GC (probably due to contention around s_csSynchProcessLock?)

cc @janvorli

[janvorli]

@vrecluse could you please capture the stack traces using lldb instead of gdb? GDB is unable to walk frames of the .NET managed code and usually a first managed frame puts it off the rails.