.NET 5 performance regression from .NET Core 3.1

[adnan-kamili]

After upgrading to ASP .NET 5.0 from 3.1 our Web API response time almost doubled from ~40ms to >80ms. The only change we made was that we upgraded the packages. I am surprised when the internet is flooded with posts about the improved performance of .NET 5.0, we saw a degraded performance.

Further technical details

• ASP.NET Core version: 5.0.1
• Runs inside a container

[mkArtakMSFT]

Thank you for filing this issue. In order for us to investigate this issue, please provide a minimalistic repro project (ideally a GitHub repo) that illustrates the problem.

[mkArtakMSFT]

To clarify, what we're looking for is a project with the minimal set of changes from the default template, which after migrating to .NET 5 will showcase the perf difference.

[adnan-kamili]

Our project is very big, so it won't be possible to provide a repo that replicates the issue. One info I can share is that we have ServerGarbageCollection disabled in the .csproj file. Can that be the issue? I read somewhere in the posts, some improvements made in server GC in .NET5

The reason for keeping it disabled is high memory usage caused over time. When ServerGC is disabled in our app, memory usage remains consistently under 400 MB. But when we enable it, it increases by around 70-100 MB every week which ultimately causes a server restart when it reaches the limit set in our K8s cluster. Our server load is around ~2 Million requests/day.

[Pilchie]

@davidfowl @sebastienros - can you give some guidance on what sort of traces would be useful here?

[sebastienros]

There are two issues here

• Memory is not bound in the limits with server GC
• Latency regression in .NET 5 on your scenario

It would be good to check if the latency is actually better with server GC on .NET 5. That would let us focus of understanding why your application's memory is not respecting K8s limits. There are ENV settings for the GC to constrain it. @adnan-kamili is that something you can try, even temporarily?

Then if you can take some traces, you can start with dotnet-trace or dotnet-monitor since it can be started as a sidecar and take a trace of a running process. Ideally one with the 3.1 and one with the 5.0 version.

[adnan-kamili]

We don't have access to 3.1 anymore as we have moved forward with .NET 5 and have made many changes since then. We actually tried server GC for few days but didn't see any change in performance. It almost remained the same.

Is there any advantage of enabling server GC?

I would be interested in ENV settings related to the server GC if it can be forced to constrain.

[brunodeuner]

today I did the tests before migrating from 3.1 to 5.0 and I noticed the same problem

in order to have a smaller project for testing, I simply created the suggested base project with the WeatherForecastController, and using jmeter I noticed a small drop in performance on requests, nothing scary, however the memory usage increased by 14%, in the case of my api which is now running in production, tests showed a more significant increase in response time and memory usage increased by almost 90%

[davidfowl]

OK I have some advice here based on our experiences with customers in this space. Performance testing is an art, spinning up performance tools on your local machines when many things are running is a bad way to test. Running client and server on the same machine, is a bad way to test, running performance tests in debug bug is a bad way to test. There are lots of gotchas and things you need to get right to do an apples to apples comparison. Local machine tests are fine when comparing the same number of requests to verify things like call counts or number of objects allocation, but not for general performance testing.

If you're interested in doing memory analysis on .NET applications then you have to read this document https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md. It's written by the .NET GC architect and has some advice on how to properly collect the right metrics to look at in order to figure out what might be happening in your application.

@adnan-kamili Running on Workstation GC is performs more poorly than server GC but is more aggressive with keeping the working set lower (in essence you've traded off CPU for Memory). Were you always using workstation GC in 3.1 or did you switch after moving to .NET 5?

@brunodeuner @adnan-kamili Are you collecting any .NET counters? It would be good to compare various metrics to see what changed between .NET 3.1 and .NET 5. That will give you 1000 ft view of things and might help us zoom in on specific areas of focus.