Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High CPU usage of application hosted as a docker container #76296

Closed
stefanolsenn opened this issue Sep 28, 2022 · 6 comments
Closed

High CPU usage of application hosted as a docker container #76296

stefanolsenn opened this issue Sep 28, 2022 · 6 comments
Labels
area-System.Threading needs-author-action An issue or pull request that requires more info or actions from the author. no-recent-activity tenet-performance Performance related issue
Milestone
Future

Comments

@stefanolsenn
Copy link

Description

We're seeing a high cpu usage - 100% on some cores on the host vm - of a application hosted in a docker container in linux. I was able to make a dotnet-dump of it, can you point me in the right direction of how to debug this?

Configuration

Using the image mcr.microsoft.com/dotnet/aspnet:6.0

Data

When running clrstack -all the last few entries are the following:

[...]

OS Thread Id: 0xbb
        Child SP               IP Call Site
00007F34267FB968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB968 00007f3ad75af457 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34267FBA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34267FBA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34267FBAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34267FBCF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34267fbcf0]
OS Thread Id: 0x23b
        Child SP               IP Call Site
00007F3403FFE968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE968 00007f3ad75af457 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3403FFEA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3403FFEA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3403FFEAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3403FFECF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3403ffecf0]
OS Thread Id: 0x240
        Child SP               IP Call Site
00007F34B0945968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945968 00007f3ad75af457 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34B0945A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34B0945A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34B0945AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34B0945CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34b0945cf0]
OS Thread Id: 0x241
        Child SP               IP Call Site
00007F3B4B6C2968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2968 00007f3ad75af457 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3B4B6C2A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3B4B6C2A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3B4B6C2AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3B4B6C2CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3b4b6c2cf0]

Analysis

We have a lot of async code in our application, we're not using Threads directly in our code only Tasks.

Could you point me in the right direction for further investigation?
@stefanolsenn stefanolsenn added the tenet-performance Performance related issue label Sep 28, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Sep 28, 2022
@ghost
Copy link

ghost commented Sep 28, 2022

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

We're seeing a high cpu usage - 100% on some cores on the host vm - of a application hosted in a docker container in linux. I was able to make a dotnet-dump of it, can you point me in the right direction of how to debug this?

Configuration

Using the image mcr.microsoft.com/dotnet/aspnet:6.0

Data

When running clrstack -all the last few entries are the following:

[...]

OS Thread Id: 0xbb
        Child SP               IP Call Site
00007F34267FB968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB968 00007f3ad75af457 [InlinedCallFrame: 00007f34267fb968]
00007F34267FB950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34267FBA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34267FBA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34267FBAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34267FBCF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34267fbcf0]
OS Thread Id: 0x23b
        Child SP               IP Call Site
00007F3403FFE968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE968 00007f3ad75af457 [InlinedCallFrame: 00007f3403ffe968]
00007F3403FFE950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3403FFEA10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3403FFEA50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3403FFEAE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3403FFECF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3403ffecf0]
OS Thread Id: 0x240
        Child SP               IP Call Site
00007F34B0945968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945968 00007f3ad75af457 [InlinedCallFrame: 00007f34b0945968]
00007F34B0945950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F34B0945A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F34B0945A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F34B0945AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F34B0945CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f34b0945cf0]
OS Thread Id: 0x241
        Child SP               IP Call Site
00007F3B4B6C2968 00007f3b51bd5ad8 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2968 00007f3ad75af457 [InlinedCallFrame: 00007f3b4b6c2968]
00007F3B4B6C2950 00007F3AD75AF457 System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32)
00007F3B4B6C2A10 00007F3ADD02F4FB System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 204]
00007F3B4B6C2A50 00007F3ADD02B6D7 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007F3B4B6C2AE0 00007F3ADD0B4D17 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 57]
00007F3B4B6C2CF0 00007f3b51240e37 [DebuggerU2MCatchHandlerFrame: 00007f3b4b6c2cf0]

Analysis

We have a lot of async code in our application, we're not using Threads directly in our code only Tasks.

Could you point me in the right direction for further investigation?

Author: stefanolsenn
Assignees: -
Labels:

area-System.Threading, tenet-performance
Milestone: -

@mangod9
Copy link
Member

mangod9 commented Sep 28, 2022

Hi @stefanolsenn, are you able to share a repro and/or dump when the issue occurs? Thx!

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Sep 28, 2022
@mangod9 mangod9 added this to the 8.0.0 milestone Sep 28, 2022
@stefanolsenn
Copy link
Author

Hi @mangod9 I got the dump file from the dotnet-dump collect command where the issue occurred - What particular output would you like from that? Thanks!

@mangod9 mangod9 added the needs-author-action An issue or pull request that requires more info or actions from the author. label Aug 9, 2023
@mangod9 mangod9 modified the milestones: 8.0.0, Future Aug 9, 2023
@ghost
Copy link

ghost commented Aug 9, 2023

This issue has been marked needs-author-action and may be missing some important information.

@ghost ghost added the no-recent-activity label Aug 23, 2023
@ghost
Copy link

ghost commented Aug 23, 2023

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

@ghost
Copy link

ghost commented Sep 7, 2023

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.

@ghost ghost closed this as completed Sep 7, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Oct 7, 2023
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Threading needs-author-action An issue or pull request that requires more info or actions from the author. no-recent-activity tenet-performance Performance related issue
Projects
None yet
Milestone
Future
Development

No branches or pull requests
2 participants
@stefanolsenn @mangod9