libcoreclr.so!EventPipeInternal::GetNextEvent high CPU use

[djluck]

Issue Title

When using EventListener over prolonged periods of time, CPU consumption slowly climbs until majority of CPU time is spent forwarding events.

General

.NET core version: v3.1.3
OS: Docker on linux (base image= mcr.microsoft.com/dotnet/core/aspnet:3.1.3-bionic)

When using the prometheus-net.DotNetRuntime package to capture CLR events, I see a gradual increase in CPU over many days:

The application itself is a simple ASP.NET core web application that does not perform intensive processing. After collecting a CPU trace using perfcollect, I see this call stack standing out by a mile:

This pattern has showed itself consistently over the last month, suggesting that the amount of CPU consumption required utilized by the event pipes increases over time to the point where we need to restart our applications.

Some notes about the prometheus-net.DotNetRuntime package (of which I authored):

• The purpose of this package is to instrument .NET runtime internals by listening to telemetry events consumed by EventListeners
• Multiple EventListener are spun up to listen to events. One listener is created per "type" of events (e.g. one for counting exceptions, one for listening to GC events, etc.)

[pcwiese]

Is there any way to mitigate this in 3.1/5.0 without tearing down the entire application? Would destroying/re-creating the EventListener's have any effect?

[josalem]

Yes, you should be able to mitigate this problem for previous versions by restarting the EventPipe session used by EventListeners periodically. For EventListeners, you should be able to call EventListener.EnableEvents with the Microsoft-Windows-DotNETRuntime provider and the same keywords as the last call to refresh the session.

[djluck]

@josalem will #48435 be merged into .NET 3.1?

[josalem]

I don't believe it will. The patch targets the C version of EventPipe that we switched to in 6.0. There is a workaround for pre-6.0 (see my previous comment on this thread).

[djluck]

Thanks @josalem. So unfortunately the workaround (Disable/ Enable events) is not always tenable due to deadlocks that occur on 3.1 when a high load of events occurs. I haven't had time to write up an issue for this yet but have a reliable reproduction on my machine. Hopefully should find the time to create an issue later today.