Memory sometimes leaks in ASP.Net Core 2.1

[pgrm]

Describe the bug

This is the memory consumption of our application over the last month. Each line is a docker container. We enforced a memory limit of 2GB and increased that to 3GB few days ago. A line stops when the container is shut down either due to a newer version being deployed or due to reaching the memory limit.

What's interesting is, that sometimes the application can maintain a stable memory consumption, while other times it's just completely out of control. The amount of requests and types of requests are fairly stable over time. We have ~6-10 requests / second split across 3 containers, so ~2-3 requests / second / container, where ~95% of the requests are the same 10 requests being polled all the time.

Here is another snapshot of the memory consumption over a longer period of time:

We tried to analyze the application using perfcollect and creating a core dump with lldb, but it didn't really help us understand the problem. Maybe we were unlucky in that we did it exactly, when the problem didn't occur.

The CPU usage of the host nodes is well below 10% and we aren't limiting it for the containers

Any idea where we could look next, or what we could try?

Thank You!

Expected behavior

To have a stable memory consumption all the time

Additional context

Our containers are based on microsoft/dotnet:2.1-aspnetcore-runtime

[muratg]

cc @sebastienros

[pgrm]

FYI, we did an update to version 2.2 and so far I didn't notice this pattern yet. So we won't be able to provide you with any additional information. I guess this issue can be closed?

[sebastienros]

Apparently I missed this issue, just saw the notification for you most recent message. I think docker limits (cgroups) is only supported by the GC since 2.2. And there are some great improvements in 3.0 regarding to this https://devblogs.microsoft.com/dotnet/using-net-and-docker-together-dockercon-2019-update/