Microsoft.AspNetCore.Server.Kestrel , Heartbeat took longer than "00:00:01"

[dingsongjie]

I ran an aspnet core service in kubernetes and was limited to 2 cpu, sometimes I found it killed by kubernetes, because its health check failed, and I also found some logs,It shows the threadpool was busy.

ServiceName:Archives @timestamp:November 22nd 2019, 12:09:11.242 LoggerName:Microsoft.AspNetCore.Server.Kestrel HostName:api-b5bdcdb89-qs649 Level:WARN Message:Heartbeat took longer than "00:00:01" at "11/22/2019 04:09:11 +00:00". _id:H4VMkW4BNxpo_f7ksKOv _type:_doc _index:service_log_2019-11-22 _score: -

But the cpu usage is low ,and I have not set a memory limit, what happend to it ?

[analogrelay]

This is almost always caused by thread-pool starvation (from things like calling .Wait()/.Result/.GetAwaiter().GetResult() on a Task). The CPU usage being low indicates that you have a lot of active threads but they're all blocked waiting on things. The Heartbeat can't run because there isn't a thread available to run on.

Our docs have some guidance on things to watch out for here: https://docs.microsoft.com/en-us/aspnet/core/performance/performance-best-practices?view=aspnetcore-3.0

Also, there's a good blog post on diagnosing thread-pool starvation specifically here: https://blogs.msdn.microsoft.com/vancem/2018/10/16/diagnosing-net-core-threadpool-starvation-with-perfview-why-my-service-is-not-saturating-all-cores-or-seems-to-stall/

Even just getting a memory dump and looking at the stacks for each thread may show what is holding up the CPU.

[dingsongjie]

Thanks for help