Memory Leak!

[aropb]

Hello all,

I found a memory leak in a simple project (.NET Core 2.2.1, OS Windows 10 Pro 1803, Build 17134.523).
Screenshots and project attached.

The problem is very serious, because the production project takes all the memory and falls.
The size of Heap Generation 2 is constantly growing!

gcserver=off kestrel

gcserver=on httpsys

gcserver=off httpsys

WebTest.zip

[sebastienros]

he production project takes all the memory and falls

• Can you explain what you mean by "falls"? Is the service actually recycled?
• How much memory is available for the application on the server?

I am running your project locally to test the behavior on my side, thanks for providing it.

[sebastienros]

More questions:

• How much load should I send to start seeing this behavior? Is running you sample "auto refresh" page enough?
• Can you try to take a dump at some point, and another one later on, ideally when you think it's running out of memory?

[aropb]

Yes, I open 3 browser tabs and see how memory grows. How best to make a dump? This is a simple example that shows the strange behavior of the server. Production is certainly a more complex project. There's a large dotMemory dump.

[muratg]

@aropb could you please answer @sebastienros's first two questions as well?

[aropb]

Hi,

The server becomes unavailable to clients. The Service must be restarted. On the server 6GB is available, the server is virtual.

Production server:

OS Name: Microsoft Windows Server 2012 R2 Standard
OS Version: 6.3.9600 N/A Build 9600
System Manufacturer: VMware, Inc.
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 45 Stepping 2 GenuineIntel ~2397 Mhz
[02]: Intel64 Family 6 Model 45 Stepping 2 GenuineIntel ~2397 Mhz

Total Physical Memory: 8 191 MB
Available Physical Memory: 6 328 MB
Virtual Memory: Max Size: 9 983 MB
Virtual Memory: Available: 8 194 MB
Virtual Memory: In Use: 1 789 MB

If the memory is all over the log file like this:

2019-01-11 11:40:22.7350 Microsoft.AspNetCore.Server.HttpSys.LogHelper.LogException Error: ListenForNextRequestAsync An operation was attempted on a nonexistent network connection at Microsoft.AspNetCore.Server.HttpSys.MessagePump.ProcessRequestsWorker() ExecutionContext.RunInternal => d__22.MoveNext => LogHelper.LogException
2019-01-11 12:28:42.4670 Microsoft.AspNetCore.Server.HttpSys.LogHelper.LogException Error: ListenForNextRequestAsync An operation was attempted on a nonexistent network connection at Microsoft.AspNetCore.Server.HttpSys.MessagePump.ProcessRequestsWorker() ExecutionContext.RunInternal => d__22.MoveNext => LogHelper.LogException
2019-01-11 12:28:42.4670 Microsoft.AspNetCore.Server.HttpSys.LogHelper.LogException Error: ListenForNextRequestAsync An operation was attempted on a nonexistent network connection at Microsoft.AspNetCore.Server.HttpSys.MessagePump.ProcessRequestsWorker() ExecutionContext.RunInternal => d__22.MoveNext => LogHelper.LogException

Dump of WebServerTest from dotMemory:
WebServerTest netcore 2.2.1 gcServer off httpsys.dmw.zip

[aropb]

The application log on the PC (Windows 10 Pro, x64, 8Gb, i7):

SDK for .NET Core:
Version: 2.2.103
Commit: 8edbc2570a

Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62

[aropb]

@sebastienros, Do you need some more information? Thanks.

[sebastienros]

No, sorry for the delay, I just need more time. Thanks.

[aropb]

Hi, @sebastienros, when can I expect an answer? Thanks.

[sebastienros]

I can't repro an issue.

I have published the application you provided with the publish.cmd script, and ran it with a much higher load than what you suggested. I also defined logging at the Warning level otherwise every requests were logged on the console. On my machine the memory is very stable at 30MB which makes sense as you disabled server GC.

Does you issue only happen through IIS in-process?

[aropb]

No, IIS i don't use.

[aropb]

Please, write your PC and OS specification.

[sebastienros]

Windows 10, .NET Core 2.2.1

[aropb]

@sebastienros, Hi.
I can not understand why my project memory is always growing. I can't find the cause, but maybe it's normal behavior?

Dump:
https://drive.google.com/open?id=1sfIJMmnHf5RQjsPr-P1nZ63iDMj7H1Ot

[sebastienros]

It might just be that gen2 GC hasn't triggered yet. Using 60MB of memory is very low, the GC will only start once it hits memory pressure, meaning the OS tells the application that it's about to use too much memory. But I might be wrong. Let it run for hours, take a memory dump (one from the task manager is fine) at the middle of the run and right before your stop it once it's taking half your system memory. Taking two dumps will let us know what objects were created during this window.

Take a look at this comment where the process takes more than 2GB and the GC is never actually called, only because it doesn't have to as the available memory is still plenty. It might not be the "expected" scenario from an application developer's perspective though, and this will be changed in .NET 3.0.

[aropb]

@sebastienros Hi, I found my bug in the controller. I'm sorry.

I used in "Select/Where" from DatabaseContext a not static method of a controller.
It's bad that the netcore compiler misses such errors.

Thanks.

[aropb]

dotnet/efcore#13048