Silo - Massive memory consumption

[johnds1974]

Hi, I'm testing the performance of a custom Silo example, which hosts a very basic Grain.
I am testing activation of 1000 unique instances of this grain to test the memory consumption of the Silo process, on Windows Server 2016. Both apps are .NET 5 console apps, the client app just runs a loop of 1000 calls to factory.GetGrain<IMyGrain>($"MyGrain-{i}");

Pretty basic.

Silo process starts off using about 62MB RAM. After all 1000 instances are 'activated', it's using 1,760MB!

This is very concerning because my architecture is to have a Grain instance per connected client, a Digital Twin if you like, and I need to have 100,000's of these overall, and I'm trying to find the most memory and hardware efficient mechanism of hosting these processes (the Silo's) on Azure without needing to spend loads on High-workload capable VM's! :-(

[ReubenBond]

You should analyze the memory usage to determine where the memory is being used. I imagine you'll find that the GC heap is much smaller than 1.7GB. It's typical for you to see memory which is allocated to the process but is not actively being used by .NET (i.e, the managed heap size is smaller). You can use a tool like Visual Studio, JetBrains dotMemory, https://memprofiler.com/, or Redgate. There are also command line tools like dotnet-dump and dotnet-gcdump.

Have you configured the process to use ServerGC or is it using workstation GC?

[johnds1974]

Hi @ReubenBond, I haven't yet gone that far to try to profile my sample Silo, will give that a try soon.

My Silo app is running the full Server GC options yes, specified in the .csproj which I took from the Orleans documentation.

Curious also is the deactivation of my grain instances, which does get invoked, actually increases memory consumption. I've even waited almost an hour to see if there's some delay to actual Garbage collection of the hopefully 'no longer reachable' instances, but to no avail. Not sure if there's something I've missed configuration wise? I've followed all the most basic of samples etc.

[ReubenBond]

The Server GC does not optimize for minimizing process memory usage (commit size) by default. You can configure some options to change that if you want.
By default it will tell Windows (using VirtualAlloc/VirtualUnlock) or Linux (using madvise) that it no longer needs the memory, but does not eagerly decommit the pages. In general, this is the right choice as it optimizes for performance.

[ReubenBond]

I suggest you experiment with Workstation GC, since Server GC uses a heap per core. You can also follow the troubleshooting guide: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/performance#issue-the-process-uses-too-much-memory

[johnds1974]

I'm still busy with this 'issue'. Trying all variations of options. Time consuming.