Kestrel dies because of Out of Memory exception

[HassanHashemi]

One of our MVC projects keeps growing memory, and it seems that Garbage collector dose not do anything with it.

The process keeps growing until it is killed by the OS.

dmesg outputs this log:

[961030.365411] Out of memory: Kill process 3673 (dotnet) score 913 or sacrifice child
[961030.365725] Killed process 3673 (dotnet) total-vm:29592364kB, anon-rss:3565416kB, file-rss:256kB, shmem-rss:0kB

What makes me think that it is related to, GC not working as expected is that, i have periodically called GC.Collect() and memory immediately drops down significantly but still keeps growing in a much slower pace.

Also GC is running under Server mode.

Environment Info

Runtime: .Net Core 2.1
ASP.NET Core 2.1
Web server: Kestrel (directly exposed)
OS: Centos 7

4 Core processor
4GB Memory

Also not sure if it any how related, i thought maybe GC dose not recognize memory limits, so here is the output of ulimit command:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15062
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15062
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Notes

• the apps works as expected on Windows server 2016.
• this may or may not be a duplicate of Asp.net Core not Collecting Garbage #1976 but all the issues in there are being tracked separately.

Update
Adding entire dmesg log here as gist to keep things clear.

[benaadams]

Related to https://github.com/dotnet/coreclr/issues/19060 ? /cc @tmds

[tmds]

Are you running this in a container? Enforcing a memory limit?

dmesg outputs this log: [961030.365725] Killed process 3673 (dotnet) total-vm:29592364kB, anon-rss:3565416kB, file-rss:256kB, shmem-rss:0kB

Is there some more info in dmesg about this?

[HassanHashemi]

@tmds No, it's not inside a container. (which is why i think cgroup is not the case here). And thus no memory limit other than OS defaults. (as far as i am aware)

Is there some more info in dmesg about this?

No, dmesg contains that one single line of log that is related to this issue.

[tmds]

Ah, I've seen dmesg OOM logging that was more verbose.

Your system may be out of swap+ram and the kernel is killing a process to free some up.

[HassanHashemi]

@tmds What do you think about GC not working?
I have deployed the app again, and this time i will also monitor swap info.

[tmds]

What do you think about GC not working?

With anon-rss:3565416kB and 4GB Memory, .NET Core still be waiting a bit before GCing.

[HassanHashemi]

@tmds Thanks for the response, any suggestion on how to fix? if that's the case.

[tmds]

Is this a machine dedicated for running Kestrel? Or does it run other applications/services?

You could install more RAM.
You can also configure the system so it becomes less likely the kernel will kill the dotnet process, but pick another instead. With the default config, probably dotnet looks like an interesting target because it is using a lot of RAM.

[HassanHashemi]

It is a dedicated machine for Kestrel.
Adding more RAM is not really an option, because the app dose not really need that much. dotnet should not be using that amount of memory anyway.
However event if i add more RAM, same thing will happen again, but a bit later, right?

[tmds]

I'm surprised no other things are running on the machine.
It's unfortunate dmesg has so little information why the process is killed.

dotnet should not be using that amount of memory anyway.

the amount of physical ram is used to determine how much memory dotnet can use.
If you want to lower it, you can run it in a cgroup.

[Drawaes]

If you run it as workstation gc it will be more agressive. How many cores is the machine because server gc has a heap per core so can easily swallow a lot of ram as it's baseline before gcing. Workstation will be more agressive (at the cost of more gc activity)

[HassanHashemi]

the amount of physical ram is used to determine how much memory dotnet can use.

as an example, with 4GB memory dotnet should not keep allocating memory, because the limit is already set by amount of physical memory, so why do we need to enforce the limit via cgroup?

[HassanHashemi]

@Drawaes Yeah, the machine has 4 Cores and is running under ServerGC. (i believe having ServerGC enabled is a reasonable thing to have)

[benaadams]

Might want to raise an issue in https://github.com/dotnet/coreclr

[Drawaes]

That is less than 1gb per core (minus os overhead) which is pretty low for the server gc.

The default GC segment sizes aren't going to be good for you.

https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals#ephemeral-generations-and-segments