asp.net core ServerGarbageCollection not working

[amirjalali1]

Asp.net core 1.1.2
windows server 2016

after migrating my project from vs 2015 to 2017 i made a release and published my website but now memory usage constantly grows in production server.

At vs 2015 i had this in my project.json :

  "runtimeOptions": {
    "gcServer": true
  },

but after migrating to vs 2017 csproj is like this

  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>
    <PreserveCompilationContext>true</PreserveCompilationContext>
    <AssemblyName>xxxxx</AssemblyName>
    <OutputType>Exe</OutputType>
    <PackageId>xxxxx</PackageId>
    <UserSecretsId>xxxxxxx</UserSecretsId>
    <RuntimeFrameworkVersion>1.1.2</RuntimeFrameworkVersion>
    <PackageTargetFallback>$(PackageTargetFallback);dotnet5.6;dnxcore50;portable-net45+win8</PackageTargetFallback>
    <ServerGarbageCollection>true</ServerGarbageCollection>
  </PropertyGroup>

i also changed csproj to this in another test:

  <PropertyGroup>
    <ServerGarbageCollection>true</ServerGarbageCollection>
    <ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
  </PropertyGroup>

and runtimeconfig.template.json to this :

{
  "gcServer": true,
  "System.GC.Server": true,
  "gcConcurrent": true
}

but garbage collection is not working. any idea what is going wrong?

[davidfowl]

How do you know it isn't working?

@Maoni0 @jkotas is there a runtime API to check what GC mode is enabled to see if configuration settings are effective?

[amirjalali1]

@davidfowl
I had a similar issue (high and growing memory usage) in vs 2015 so setting "gcServer" to true solved the problem.
And now the only thing that has changed in my project is migrating from vs 2015 project.json to vs 2017 csproj.

the app was working without high memory usage from the previous release until now which i migrated to vs 2017 and publish the app again.

before migration the memory usage was around 400 Mb

and now its around 1685904 and growing
and i ran jetbrains dotmemory on development everything works fine between memory snapshots
and even on vs memory snapshots
also i have an api checking these two vars

        var latencyMode = System.Runtime.GCSettings.LatencyMode;
            var isServerGC = System.Runtime.GCSettings.IsServerGC;

  latencymode is  Interactive isServerGC is True      

is there anything else i can check ?

How do i get gc type on production?

I even made a memory dump its around 2.5 Gb and i didn't find anything suspicious

[Maoni0]

Interactive and IsServerGC being true means you are using Concurrent and Server GC. Sounds like there are 2 issues -

1. Locally you are seeing high memory usage again after upgrading from VS2015 to VS2017 (and presumably you were using Concurrent and Server GC when you were using VS2015). Do you happen to have some comparison perf data? eg, process dumps, ETW traces?

2. On production, you'd like to find out if you are using Concurrent and Server GC. If you don't have the option to add these 2 variables, do you have the option to take a process dump?

[amirjalali1]

@Maoni0

1. locally (on the dev env) i don't see any high memory usage only on production and on the sever. on dev i didn't make any process dump but on production i made a dump after this issue the dump file size is around 2.25 Gb.
2. on production i have added these two vars and "latencymode Interactive and isServerGC True" is on production. yes on production i have made process dump. what to look for?

[Maoni0]

ahh, so I totally misunderstood you then. The 2 vars you added are enough to confirm that you are using Concurrent Server GC. Note that the first mode (Interactive) can be changed by some code but the Server GC part is not changeable after the process starts. You can confirm by looking at callstacks in the process dump you took and you should see a few callstacks that have SVR::gc_heap::gc_thread_function on them. if SVR::gc_heap::settings.pause_mode is pause_interactive, it means you are using concurrent.

[amirjalali1]

@Maoni0
I did stress test on dev env today the memory usage didn't go further than 300Mb but on production memory usage constantly growing

[amirjalali1]

@Maoni0
here is a memory dump

[Maoni0]

@swgillespie could you please help @1amirjalai with doing some perf investigation here? thanks!

[swgillespie]

@1amirjalai

For troubleshooting GC issues I'd definitely recommend using PerfView: https://github.com/Microsoft/perfview#getting-perfview . With PerfView you can analyze your heap dumps or capture a live trace where you can see why GCs are happening (or not happening). With your scenario running, you can collect a GC trace using the command

PerfView.exe /GCCollectOnly /nogui /NoV2Rundown /NoNGENRundown /NoRundown /merge:true /zip:true collect

You can then open this trace in PerfView and look at the "GcStats" view (see here for an example). If GCs are occuring, you'll see in this view where they occurred and what triggered them. I'd definitely recommend you taking a trace like this; from there we can take a look and see what's going on in your process.

Without any data, though, I can say that in general Server GC will favor latency and throughput over heap size. If the GC thinks that the machine has enough memory, it will feel more free to allow the heap to expand in order to avoid compaction. If you want to limit the memory consumption of your application, you can use job objects to do so and the GC will respect the limits that you set, compacting the heap if your memory usage approaches the job object limit.

[amirjalali1]

@Maoni0 @swgillespie here are the results of :

PerfView.exe /GCCollectOnly /nogui /NoV2Rundown /NoNGENRundown /NoRundown /merge:true /zip:true collect

The app source code hasn't changed from project.json to csproj but memory consumption has changed a lot.
even dot.net version is the same.

has gc behavior changed from project.json to csproj?

[swgillespie]

@1amirjalai Apologies for the late response, I just found out that GitHub failed to deliver this notification to me because of some obscure settings I have set... I'm taking a look at this today and I'll write a response this afternoon.

[swgillespie]

@1amirjalai

It looks like there's a period of 40 seconds in which your application allocates 1GB in Gen 0 and doesn't take a GC in that time. I see that you have at least 24 heaps; how many cores does this machine have?

Server GC aggressively parallelizes across all cores that you have and creates a logical heap for every core. Your allocation rate of 28 mb/s is quite low given the number of cores you have on your machine. Since each one of these heaps has a (quite high) gen 0 budget, and you are allocating at only 28 mb/s, it's not surprising that it would take so long for a GC to be triggered - Server GC does not believe one is necessary due to the preponderance of resources available to it (both memory and CPU time).

has gc behavior changed from project.json to csproj?

No, the GC operates at a lower level of abstraction than the CLI and is not aware of project.json or csproj, or even how the application is run. However, the speed in which Gen 0 collections are triggered is dependent entirely on the gen 0 budget size for each of the various server GC heaps, and this number is very sensitive to your environment.

Can you show a trace of this same process where you see that the memory usage is lower, so we can compare these two traces?

[amirjalali1]

@swgillespie sorry for late response i was doing some reading on the windows internals.

The system i'm using has 24 cores and 256 Gb of ram.

Can you show a trace of this same process where you see that the memory usage is lower, so we can compare these two traces?

I wasn't able to open previous dump files using perfview for getting gc info.

[amirjalali1]

@swgillespie Today i also migrated and made a publish of our another small web api app which we hadn't touched for a long time just for testing to examine memory usage consumption.

Again after migrating to vs2017 and publishing the app without any modifying of code lines the memory usage grows constantly. it's a very small app but memory consumption is very high.

I also made another dump today of my main app and i don't know it's related to this issue or not but now i'm having a deadlock

OS Thread Id: 0x18e68 (27)
        Child SP               IP Call Site
000000e1b8bfeeb8 00007ffa506f6c24 [GCFrame: 000000e1b8bfeeb8] 
000000e1b8bfef88 00007ffa506f6c24 [HelperMethodFrame_1OBJ: 000000e1b8bfef88] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
000000e1b8bff0b0 00007ff9d51c36c1 StackExchange.Redis.SocketManager.WriteAllQueues()
000000e1b8bff100 00007ff9d51c3407 StackExchange.Redis.SocketManager+c.b__27_0(System.Object)
000000e1b8bff140 00007ffa1c5b16ee System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000e1b8bff390 00007ffa3490cb13 [GCFrame: 000000e1b8bff390] 
000000e1b8bff668 00007ffa3490cb13 [DebuggerU2MCatchHandlerFrame: 000000e1b8bff668] 

[swgillespie]

@1amirjalai

Unfortunately there's not a whole lot we can be sure of without a trace, but running witih server GC on a machine that powerful results in a GC that is capable of servicing a high (on the order of GB/s) allocation rates, and is tuned to that scenario. If you don't allocate much, you won't GC much. Server GC generally runs under the assumption that it has full control over all cores and memory of the machine and, as such, it won't GC if it doesn't think it needs to (and on a machine that huge, there's hardly any reason to - if there's memory to use, might as well use it).

If you want to limit the amount of memory that your process uses, you can consider using a job object. If you put your process in a job object and limit the job object's memory to something lower than the amount of RAM on the machine, the GC will respect that limit and GC more aggressively. At the moment, there's no way to ask the GC to be more frugal with memory; there have been some issues on the CoreCLR repo discussing this as a possible feature for the future, though.