Dynamically resize stream info caches based on available memory

[shaan1337]

Added: Dynamically resize stream info caches based on available memory

This PR adds support for caches that can be automatically resized based on free memory. Right now only one cache is dynamic: the StreamInfo cache but others can be added later on. A user can still make the cache static by specifying --stream-info-cache-capacity.

The rationale behind the changes is that a static cache which is scaled to free memory available on startup would not be able to efficiently deal with these scenarios:

• growth of the database with time
• heavy operations like index merges, scavenging etc. (an estimate could be done but it would waste memory during non-peak hours)
• launching multiple db instances on a single machine (not recommended)
• growth in the number of clients connected
• spike in memory usage of other processes on the system

Memory allocation ratio

The StreamInfo cache is a logical cache comprised of the LastEventNumber and StreamMetadata cache. Currently 60% of the memory allocated to the logical cache is reserved to the LastEventNumber cache and 40% to StreamMetadata cache (this is based on empirical testing).

Cache resizer hierarchy

The cache resizers are modeled as a tree structure. When resizing caches, we specify the capacity that we wan't to allocate to the root of the tree. The children (and sub-children) will then derive their capacities based on their weights and the amount allocated to them by their parent.

Resizing algorithm

The resizing algorithm is as follows:

• Every 15 seconds:
  • first check if the amount of memory available for caching has changed by more than 200MB since the last resize. If this condition is true, continue to the next step, otherwise don't do anything.
  • check if there's less than 25% or 6 GiB free memory. If this condition is true, resize the caches, otherwise go to next step. The total free memory takes these factors into consideration
    • free system memory
    • free memory on the .NET heap
    • memory deleted by the caches but not yet garbage collected.
  • check if 10 minutes has passed since the last resize. If condition is true, resize caches, otherwise do nothing.

The 200 MB change threshold ensures that:

• We don't resize too often if not much has changed.

Resizing the caches every 10 mins ensures that:

• the caches can grow again when more memory is available
• we don't abruptly decrease the cache sizes by a large value (unless there's an abrupt free memory increase within past 10 minutes)

Checking every 15 seconds ensures that:

• we can quickly act if we exceed the 25% / 6 GiB free memory threshold before it's too late and we run out of memory

New stats

The following stats were added under .stats.es:

  "cache": {
    "StreamInfo": {
      "LastEventNumber": {
        "name": "LastEventNumber",
        "count": 266339,
        "sizeBytes": 45833588,
        "capacityBytes": 2032332494,
        "utilizationPercent": 2.2552209412245907
      },
      "Metadata": {
        "name": "Metadata",
        "count": 133169,
        "sizeBytes": 34591868,
        "capacityBytes": 1354888329,
        "utilizationPercent": 2.553115799995942
      },
      "name": "StreamInfo",
      "count": 399508,
      "sizeBytes": 80425456,
      "capacityBytes": 3387220824,
      "utilizationPercent": 2.3743788840145603
    },
    "name": "cache",
    "count": 399508,
    "sizeBytes": 80425456,
    "capacityBytes": 3387220824,
    "utilizationPercent": 2.3743788840145603
  }

.stats.sys.totalMem - total system memory

.stats.proc.gc.fragmentation - what percent of the heap is fragmented

TODOs in later PRs

• Convert the chunks cache to a dynamic cache (e.g if ChunksCacheSize is set to -1)
• Add a StreamInfoCacheSize parameter (measured in bytes) which deprecates StreamInfoCacheCapacity (breaking change)

[shaan1337]

Based on tests with a large instance, it seems that the size calculation is lower by ~25% than the actual size, i'll update the PR to make the calculation more precise (LRU cache overhead is not properly accounted for)

[timothycoleman]

i pushed a bunch of trivial adjustments to a branch dynamic-cache-sizeing-tim-suggestions, consider these suggestions you can merge/squash in whichever ones you like

[shaan1337]

hey @timothycoleman thanks for the feedback! will go through it asap.

i've pushed a couple more commits for more precise calculation and also some fixes (where memory wasn't being properly released)

[hayley-jean]

Will this need to be rebased due to the dotnet6 upgrade?