High CPU usage of application hosted as a docker container

[stefanolsenn]

Description

We're seeing a high cpu usage - 100% on some cores on the host vm - of a application hosted in a docker container in linux. I was able to make a dotnet-dump of it, can you point me in the right direction of how to debug this?

Configuration

Using the image mcr.microsoft.com/dotnet/aspnet:6.0

Data

When running clrstack -all the last few entries are the following:

[...]

OS Thread Id: 0xbb
        Child SP               IP Call Site
00007F34267FB968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB968 00007f3ad75af457 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34267FBA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34267FBA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34267FBAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34267FBCF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34267fbcf0]
OS Thread Id: 0x23b
        Child SP               IP Call Site
00007F3403FFE968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE968 00007f3ad75af457 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3403FFEA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3403FFEA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3403FFEAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3403FFECF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3403ffecf0]
OS Thread Id: 0x240
        Child SP               IP Call Site
00007F34B0945968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945968 00007f3ad75af457 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34B0945A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34B0945A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34B0945AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34B0945CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34b0945cf0]
OS Thread Id: 0x241
        Child SP               IP Call Site
00007F3B4B6C2968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2968 00007f3ad75af457 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3B4B6C2A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3B4B6C2A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3B4B6C2AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3B4B6C2CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3b4b6c2cf0]

Analysis

We have a lot of async code in our application, we're not using Threads directly in our code only Tasks.

Could you point me in the right direction for further investigation?

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We're seeing a high cpu usage - 100% on some cores on the host vm - of a application hosted in a docker container in linux. I was able to make a dotnet-dump of it, can you point me in the right direction of how to debug this?

Configuration

Using the image mcr.microsoft.com/dotnet/aspnet:6.0

Data

When running clrstack -all the last few entries are the following:

[...]

OS Thread Id: 0xbb
        Child SP               IP Call Site
00007F34267FB968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB968 00007f3ad75af457 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34267FBA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34267FBA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34267FBAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34267FBCF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34267fbcf0]
OS Thread Id: 0x23b
        Child SP               IP Call Site
00007F3403FFE968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE968 00007f3ad75af457 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3403FFEA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3403FFEA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3403FFEAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3403FFECF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3403ffecf0]
OS Thread Id: 0x240
        Child SP               IP Call Site
00007F34B0945968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945968 00007f3ad75af457 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34B0945A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34B0945A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34B0945AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34B0945CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34b0945cf0]
OS Thread Id: 0x241
        Child SP               IP Call Site
00007F3B4B6C2968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2968 00007f3ad75af457 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3B4B6C2A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3B4B6C2A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3B4B6C2AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3B4B6C2CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3b4b6c2cf0]

Analysis

We have a lot of async code in our application, we're not using Threads directly in our code only Tasks.

Could you point me in the right direction for further investigation?

Author:	stefanolsenn
Assignees:	-
Labels:	area-System.Threading, tenet-performance
Milestone:	-

[mangod9]

Hi @stefanolsenn, are you able to share a repro and/or dump when the issue occurs? Thx!

[stefanolsenn]

Hi @mangod9 I got the dump file from the dotnet-dump collect command where the issue occurred - What particular output would you like from that? Thanks!

This issue has been marked needs-author-action and may be missing some important information.

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.