Memory not releasing effectively: WorkingSet keeps rising despite small dataset

[mgzhenhong]

Context:

I am using Garnet v1.1.1 with C# .NET 8.0 to create a basic Garnet Server for performance testing.
After the server starts, it performs an HMSet operation on 3 keys every 100ms. Each key contains 100 fields, and all values are random numbers.
During the run, I log the process's Environment.WorkingSet and the output of server.Metrics.GetInfoMetrics(Garnet.common.InfoMetricsType.MEMORY).
Even in this minimal environment, Environment.WorkingSet starts at approximately 60MB and keeps rising to over 700MB. It then suddenly drops back to around 500MB, only to rise again. Each subsequent peak is slightly higher than the previous one.
Below are the startup configuration I used and the log showing one instance of this sudden memory drop.
I don't understand why such a small dataset requires so much memory. Are there other configuration options I need to set to keep memory usage stable?

string[] startArgs = [
    $"--port={ServiceHelper.ProjectConfig.ServicePortInfo.CacheServerTcpPort}",
    $"--bind={bindIp}",
    "--storage-tier=false",
    "--aof=false",
    "--recover=false",
];
this.server = new GarnetServer(startArgs);

the memory key changes in log as below:

15:21:15 | INFO | Process Memory used: 724.78 MB
15:21:15 | INFO | # Garnet Memory Info:
15:21:15 | INFO |     system_page_size: 4096
15:21:15 | INFO |     total_system_memory: 10736967680
15:21:15 | INFO |     total_system_memory(MB): 10239
15:21:15 | INFO |     available_system_memory: 4574695424
15:21:15 | INFO |     available_system_memory(MB): 4362
15:21:15 | INFO |     proc_paged_memory_size: 771407872
15:21:15 | INFO |     proc_paged_memory_size(MB): 735
15:21:15 | INFO |     proc_peak_paged_memory_size: 772702208
15:21:15 | INFO |     proc_peak_paged_memory_size(MB): 736
15:21:15 | INFO |     proc_pageable_memory_size: 1234976
15:21:15 | INFO |     proc_pageable_memory_size(MB): 1
15:21:15 | INFO |     proc_private_memory_size: 771407872
15:21:15 | INFO |     proc_private_memory_size(MB): 735
15:21:15 | INFO |     proc_virtual_memory_size: 2483828973568
15:21:15 | INFO |     proc_virtual_memory_size(MB): 2368763
15:21:15 | INFO |     proc_peak_virtual_memory_size: 2483857088512
15:21:15 | INFO |     proc_peak_virtual_memory_size(MB): 2368790
15:21:15 | INFO |     proc_physical_memory_size: 760012800
15:21:15 | INFO |     proc_physical_memory_size(MB): 724
15:21:15 | INFO |     proc_peak_physical_memory_size: 760250368
15:21:15 | INFO |     proc_peak_physical_memory_size(MB): 725
15:21:15 | INFO |     gc_committed_bytes: 647249920
15:21:15 | INFO |     gc_heap_bytes: 637161200
15:21:15 | INFO |     gc_managed_memory_bytes_excluding_heap: 10088720
15:21:15 | INFO |     gc_fragmented_bytes: 37859256
15:21:15 | INFO |     main_store_index_size: 131072
15:21:15 | INFO |     main_store_log_memory_size: 8192
15:21:15 | INFO |     main_store_read_cache_size: 0
15:21:15 | INFO |     total_main_store_size: 139264
15:21:15 | INFO |     object_store_index_size: 1048576
15:21:15 | INFO |     object_store_log_memory_size: 8192
15:21:15 | INFO |     object_store_heap_memory_target_size: 1048576
15:21:15 | INFO |     object_store_heap_memory_size: 18272
15:21:15 | INFO |     object_store_read_cache_log_memory_size: 0
15:21:15 | INFO |     object_store_read_cache_heap_memory_size: 0
15:21:15 | INFO |     total_object_store_size: 1075040
15:21:15 | INFO |     aof_memory_size: -1
15:21:15 | INFO | # Garnet Memory Info END


15:22:15 | INFO | Process Memory used: 505.71 MB
15:22:15 | INFO | # Garnet Memory Info:
15:22:15 | INFO |     system_page_size: 4096
15:22:15 | INFO |     total_system_memory: 10736967680
15:22:15 | INFO |     total_system_memory(MB): 10239
15:22:15 | INFO |     available_system_memory: 4829552640
15:22:15 | INFO |     available_system_memory(MB): 4606
15:22:15 | INFO |     proc_paged_memory_size: 772026368
15:22:15 | INFO |     proc_paged_memory_size(MB): 736
15:22:15 | INFO |     proc_peak_paged_memory_size: 772702208
15:22:15 | INFO |     proc_peak_paged_memory_size(MB): 736
15:22:15 | INFO |     proc_pageable_memory_size: 1235168
15:22:15 | INFO |     proc_pageable_memory_size(MB): 1
15:22:15 | INFO |     proc_private_memory_size: 772026368
15:22:15 | INFO |     proc_private_memory_size(MB): 736
15:22:15 | INFO |     proc_virtual_memory_size: 2483832119296
15:22:15 | INFO |     proc_virtual_memory_size(MB): 2368766
15:22:15 | INFO |     proc_peak_virtual_memory_size: 2483857088512
15:22:15 | INFO |     proc_peak_virtual_memory_size(MB): 2368790
15:22:15 | INFO |     proc_physical_memory_size: 530358272
15:22:15 | INFO |     proc_physical_memory_size(MB): 505
15:22:15 | INFO |     proc_peak_physical_memory_size: 760250368
15:22:15 | INFO |     proc_peak_physical_memory_size(MB): 725
15:22:15 | INFO |     gc_committed_bytes: 647974912
15:22:15 | INFO |     gc_heap_bytes: 637387864
15:22:15 | INFO |     gc_managed_memory_bytes_excluding_heap: 10587048
15:22:15 | INFO |     gc_fragmented_bytes: 29950408
15:22:15 | INFO |     main_store_index_size: 131072
15:22:15 | INFO |     main_store_log_memory_size: 8192
15:22:15 | INFO |     main_store_read_cache_size: 0
15:22:15 | INFO |     total_main_store_size: 139264
15:22:15 | INFO |     object_store_index_size: 1048576
15:22:15 | INFO |     object_store_log_memory_size: 8192
15:22:15 | INFO |     object_store_heap_memory_target_size: 1048576
15:22:15 | INFO |     object_store_heap_memory_size: 18264
15:22:15 | INFO |     object_store_read_cache_log_memory_size: 0
15:22:15 | INFO |     object_store_read_cache_heap_memory_size: 0
15:22:15 | INFO |     total_object_store_size: 1075032
15:22:15 | INFO |     aof_memory_size: -1
15:22:15 | INFO | # Garnet Memory Info END

[Vasileios Zois (vazois)]

$envVars = @{
"DOTNET_gcServer" = "1"
"DOTNET_GCConserveMemory" = "9"
"DOTNET_GCHeapCount" = "c"
"DOTNET_gcServer" = "0"
}

Try using a combination of this environment variables to force the GC to aggressively reclaim memory.