-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libcoreclr.so!EventPipeInternal::GetNextEvent high CPU use #43985
Comments
Is there any way to mitigate this in 3.1/5.0 without tearing down the entire application? Would destroying/re-creating the EventListener's have any effect? |
Yes, you should be able to mitigate this problem for previous versions by restarting the EventPipe session used by |
I don't believe it will. The patch targets the C version of EventPipe that we switched to in 6.0. There is a workaround for pre-6.0 (see my previous comment on this thread). |
Thanks @josalem. So unfortunately the workaround (Disable/ Enable events) is not always tenable due to deadlocks that occur on 3.1 when a high load of events occurs. I haven't had time to write up an issue for this yet but have a reliable reproduction on my machine. Hopefully should find the time to create an issue later today. |
Issue Title
When using
EventListener
over prolonged periods of time, CPU consumption slowly climbs until majority of CPU time is spent forwarding events.General
.NET core version: v3.1.3
OS: Docker on linux (base image=
mcr.microsoft.com/dotnet/core/aspnet:3.1.3-bionic
)When using the prometheus-net.DotNetRuntime package to capture CLR events, I see a gradual increase in CPU over many days:
![image](https://user-images.githubusercontent.com/1540536/97253371-55c78c00-1860-11eb-8d70-d96342735172.png)
The application itself is a simple ASP.NET core web application that does not perform intensive processing. After collecting a CPU trace using
![image](https://user-images.githubusercontent.com/1540536/97253435-81e30d00-1860-11eb-8f5f-e3d126124e67.png)
perfcollect
, I see this call stack standing out by a mile:This pattern has showed itself consistently over the last month, suggesting that the amount of CPU consumption required utilized by the event pipes increases over time to the point where we need to restart our applications.
Some notes about the prometheus-net.DotNetRuntime package (of which I authored):
EventListener
sEventListener
are spun up to listen to events. One listener is created per "type" of events (e.g. one for counting exceptions, one for listening to GC events, etc.)The text was updated successfully, but these errors were encountered: