Memory usage reported by the process greatly mismatches the dump

[Leonardo-Ferreira]

Description

I just caught wind of a memory leak in my app and of course this is happening in production... here is what It looks like:

AKS and Application Insights both report a very close memory usage:

The thing is: when I got into that pod and extracted the dump, the dump only accounted for like 30mb!, take a look:

Where did the memory go? is this a bug on the dump? on the analyzer?

Configuration

.NET 7, lastest version as of today
OS is the one in the official asp.net core docker img
x64
The pod is 1GB ram and 1cpu

Regression?

Never had this problem before

Data

No extra data now, but feel free to ask!

Analysis

Might be related to #80351 ?!?

[jkotas]

Where did the memory go? is this a bug on the dump? on the analyzer?

The analyzer analyzes GC heap memory only. It means that the leak is most likely in unmanaged memory.

@tommcdon This looks like the perfmap memory leak that you have been looking into. (Notice that the screen snapshot shows number of finalizable dynamic methods. It means that this workload keeps churning through JITed code that is causing the leak.)

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I just caught wind of a memory leak in my app and of course this is happening in production... here is what It looks like:

AKS and Application Insights both report a very close memory usage:

The thing is: when I got into that pod and extracted the dump, the dump only accounted for like 30mb!, take a look:

Where did the memory go? is this a bug on the dump? on the analyzer?

Configuration

.NET 7, lastest version as of today
OS is the one in the official asp.net core docker img
x64
The pod is 1GB ram and 1cpu

Regression?

Never had this problem before

Data

No extra data now, but feel free to ask!

Analysis

Might be related to #80351 ?!?

Author:	Leonardo-Ferreira
Assignees:	-
Labels:	tenet-performance, area-Diagnostics-coreclr, untriaged, needs-area-label
Milestone:	-

[Leonardo-Ferreira]

out of curiosity: how can I verify that the memory is indeed on unmanaged sections?

[hoyosjs]

A possible thing that could let you know that - if you have a full memory elf core and access to a windows machine - is to use Windbg and the latest release of SOS. the command !sos maddress should show what type of pages are getting used. !dumpheap -stat -live and !dumpheap -stat -dead could also give you an idea if there's GC memory that hasn't been collected and might be rooting anything.

[tommcdon]

Hello @Leonardo-Ferreira please let us know if !sos maddress shows where the extra memory is going, and if there is any further action on our part for this issue.

This issue has been marked needs-author-action and may be missing some important information.