Full GC starts every 24 hours after system starting

[nsinitsyn]

We have migrated our project from .NET Framework 4.8 to .NET 7. After this we have got problem with long running full GC every 24 hours after system starting. We have information about GC reason from ETW - "Induced". But we haven't any explicit GC calls in out code or code of external libraries. Also we have tried to revert GC mode to .NET 6 by this article: https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21 but this solution didn't solve.

Our question:
Is there any code in .NET 7 or GC that can call full gc every 24 hours after system starting?

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have migrated our project from .NET Framework 4.8 to .NET 7. After this we have got problem with long running full GC every 24 hours after system starting. We have information about GC reason from ETW - "Induced". But we haven't any explicit GC calls in out code or code of external libraries. Also we have tried to revert GC mode to .NET 6 by this article: https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21 but this solution didn't solve.

Our question:
Is there any code in .NET 7 or GC that can call full gc every 24 hours after system starting?

Author:	nsinitsyn
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[Maoni0]

from https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-see-why-a-gc-decided-to-collect-a-generation -

The ones that almost always raise a red flag are the Induced ones as those mean some code actually is triggering GCs on its own. We have a GCTriggered event specifically for finding out what code triggered a GC with its callstack. You can collect a very lightweight trace with just the GC informational level with stack and minimal kernel events:

PerfView.exe /nogui /accepteula /KernelEvents=Process+Thread+ImageLoad /ClrEvents:GC+Stack /ClrEventLevel=Informational /BufferSize:3000 /CircularMB:3000 collect

And then look at the stacks for the GCTriggered events in Any Stacks view: 

there's a pic in the doc that shows an example

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have migrated our project from .NET Framework 4.8 to .NET 7. After this we have got problem with long running full GC every 24 hours after system starting. We have information about GC reason from ETW - "Induced". But we haven't any explicit GC calls in out code or code of external libraries. Also we have tried to revert GC mode to .NET 6 by this article: https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21 but this solution didn't solve.

Our question:
Is there any code in .NET 7 or GC that can call full gc every 24 hours after system starting?

Author:	nsinitsyn
Assignees:	-
Labels:	tenet-performance, area-GC-coreclr
Milestone:	-

[nsinitsyn]

Hello @Maoni0 , thank you for your answer.

I caught GCTriggered event in Perf View and looks like it's method ForceGCForDiagnostics. I don't find any information about this - why this function call every 24 hours and more important - how can I disable this calling?

[nsinitsyn]

I found the article with similar call stack: https://devblogs.microsoft.com/dotnet/you-should-never-see-this-callstack-in-production. According to this article method ForceGCForDiagnostics means someone requested to walk the heap with ETW. It means someone ask memory dump? Is there any way to find out who is?

[danmoseley]

What did you find out @nsinitsyn

[nsinitsyn]

We use Prometheus.DotNetRuntime for collecting metrics (https://github.com/djluck/prometheus-net.DotNetRuntime). And he recreates all ETW event listeners every 24 hours by default settings.
Calling a EventListener.Dispose results in full scan of the heap (this can be seen in the stack screenshot above). I don't know why. This is strange behavior in my opinion. Anyway we turned off recreating event listeners and it solve our problem.