memory leak during the use of reloadOnChange: true with ConfigurationBuilder

[Eilon]

From @jkotas on July 23, 2018 8:12

From @abelpalaty on July 23, 2018 7:39

Hi,
We have dotnet core driver which connects to our product (CICS TG).
In the driver we create a connection and closes it after each transaction.
While creating the connection we have the following code.

     var  builder = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);

With the setting reloadOnChange: true we see the memory leak with time eventhough the connection is closed after each request.

We tried with the following code

     var  builder = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", optional: false, reloadOnChange: false);

with the setting reloadOnChange: false we see the memory usage is constant with time.

The sample we ran runs same application in a loop of like 2500 times where each time new object is created and closed.
Our customer reported this issue and its recreating every time. Its high priority .
Is there any fix available already or is it a known issue?
Is there a way we can make sure memory usage is constant (maybe additional code to release FileSystemWatcher after the connection is closed.)

We created a memory profiling report and we see that the function which allocated the most memory is System.IO.FileSystemWatcher.AllocateBuffer().
I am attaching the screenshots of the observation we did during our testing.

Copied from original issue: dotnet/coreclr#19080

Copied from original issue: dotnet/aspnetcore#3351

[Eilon]

@abelpalaty have you noticed a long-term trend of memory growth? I think it's possible that within only 30 seconds that not all unused memory got cleaned up by the .NET Garbage Collector. The GC algorithm takes many factors into account, and with the overall memory usage of the process being fairly low (less than 80MB), the GC might not sense enough pressure to warrant collecting.

[abelpalaty]

Hi,
The customer had tried long term around 100000 requests and they had seen memory growth and reported it.
I did for just 2500 requests and were able to see same pattern.
In the above attached screenshots we see a GC on process memory where a orange mark is there.
In case of true we dont see anychange but increase of memory. While in case on false with GC we see a small decrease in memory usage while remains almost constant throughout.

The increase in objects and heap size is also huge in case of true.. Is that expected for a change of reloadOnChange: true.
Thanks

[Eilon]

I wonder if this issue is related to the discussion at dotnet/aspnetcore#1976 ...

[abelpalaty]

@Eilon
I went through Asp.net Core not Collecting Garbage.
But in this case we are specific to reloadonchange. I dont see anything connecting to this.
Thanks

[DamianEdwards]

@ajcvickers shall we have someone attempt to repro this one?

[ajcvickers]

@HaoK would be that someone.

[HaoK]

@ajcvickers I think the issue is that we never dispose the initial change token: https://github.com/aspnet/Configuration/blob/master/src/Config.FileExtensions/FileConfigurationProvider.cs#L32

Normally this isn't an issue for the main application configuration file. But in the scenario where they are constantly creating new configuration builders, each configuration is leaking the initial file watcher which explains the behavior above. I'll prep a fix and see if that takes care of the issue

[HaoK]

So one complication is there is no explicit notion of disposing config right now, so technically other than if gc kicks in, there's no way to signal that you are done with a configuration file to trigger actually closing the reload on change file watcher...

[sebastienros]

Might be related to https://github.com/aspnet/Home/issues/3110

[HaoK]

Yeah looks like the same issue