Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase in unmanaged memory growth when upgrading from .NET 6 to 8 #109933

Open
seanamorosoamtote opened this issue Nov 18, 2024 · 2 comments
Open
Labels
area-VM-coreclr tenet-performance Performance related issue untriaged New issue has not been triaged by the area owner

Comments

@seanamorosoamtote
Copy link

seanamorosoamtote commented Nov 18, 2024

Description

Issue is very similar to #95922 however we have upgraded all the way to the current release of 8.0 as of 11/18/2024 which is 8.0.10. Noticed that we started getting OOMKilled on our deployments with linux/amd64 using the base image of mcr.microsoft.com/dotnet/sdk:8.0.11. We had defaulted our memory limits to 512Mi which was sufficient for these services in dotnet 6. Now we have to update to the memory limit to 1Gi which seems to be sufficient but still can't understand where the waste is going.

Analysis seems to show that a larger deal of unmanaged memory is being reserved and not garbage collected or freed.

Configuration

Deployed in Google Cloud Kubernetes using base image of mcr.microsoft.com/dotnet/sdk:8.0.11

Regression?

Maybe? If nothing else, our application requires more base memory than it did in dotnet 6 and being able to understand why would be helpful.

Data

Prior to analyzing we would deploy with 512Mi and once our application would start the integration test phase where it would be "live" it would then get OOMKilled. Again, putting it up to 1Gi seems to resolve the issue but it seems odd that we would have to increase this just for dotnet 8.

Heaptrack analysis (notice the peak RSS is 1.1GB):
Image

There is some leakage here but nothing that seems to point to 1GB:
Image

Having an issue attaching the heaptrack log itself. I'll try after this is created to do so again.

WinDbg shows a large amount in the PAGE_READWRITE

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unknown>                              4081     7fff`f560ee00 ( 128.000 TB) 100.00%  100.00%
Image                                   465        0`0a902200 ( 169.008 MB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        938     7fff`a5346000 ( 127.999 TB)          100.00%
MEM_PRIVATE                            3608        0`5abcb000 (   1.418 GB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        938     7fff`a5346000 ( 127.999 TB) 100.00%  100.00%
MEM_COMMIT                             3608        0`5abcb000 (   1.418 GB)   0.00%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1741        0`4b21a000 (   1.174 GB)   0.00%    0.00%
PAGE_READONLY                           550        0`08c63000 ( 140.387 MB)   0.00%    0.00%
PAGE_EXECUTE_READ                      1317        0`06d4e000 ( 109.305 MB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown>                                 0`00000000     55c4`13519000 (  85.766 TB)
Image                                  7c86`6c522000        0`01dcf000 (  29.809 MB)

In WinDbg, I also looked at the largest items on the heap so this seems to point to more of a native memory issue?

55c41473fb20     964  1,267,128 Free
7c87a090bf68   7,117  1,457,672 System.Object[]
7c87a24aa700  20,236  1,618,880 System.Signature
7c87a0909db8  58,446  2,337,840 System.RuntimeType
7c87a228b390  26,386  2,744,144 System.Reflection.RuntimeMethodInfo
7c87a902cee8  17,787  2,988,216 Google.Protobuf.Reflection.FieldDescriptorProto
7c87a18e1210  22,855  4,996,152 System.Byte[]
7c87a09bd7c8 181,517 16,487,958 System.String
Total 970,322 objects, 70,404,912 bytes
@seanamorosoamtote seanamorosoamtote added the tenet-performance Performance related issue label Nov 18, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Nov 18, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Nov 18, 2024
Copy link
Contributor

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

@mangod9
Copy link
Member

mangod9 commented Nov 25, 2024

To check if it's related to the new GC in 8.0, can you try with DOTNET_GCName=libclrgc.so to determine if with that usage stays closer to 6?

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-VM-coreclr tenet-performance Performance related issue untriaged New issue has not been triaged by the area owner
Projects
None yet
Development

No branches or pull requests

4 participants