Dynamically resize cache depending on the amount of RAM available

[petethepig]

Introduction

Pyroscope keeps most frequently used objects in an LFU cache layer in RAM. This way we get good write and read performance.

See our Storage design doc for more context on the overall architecture.

Summary of the problem

Currently the size of this cache is hardcoded at 1,000 or 10,000 depending on the type of object, see the relevant code here.

This works for an average system with ~ 8GB of RAM, but fails if you want to use pyroscope on a smaller or larger machine. With smaller ones you end up with out-of-memory (OOM) events, and on larger systems you're just not utilizing all of RAM available, which is also bad.

When cache gets too big it takes time to save it to disk

Proposed solution

We need to change the way these caches work. We need to make the cache take into consideration the amount of RAM available. Currently we use this project https://github.com/dgrijalva/lfu-go for LFU implementation. We should probably fork it or write our own.

TODO:

• modify LFU cache to evict based on amount of RAM available, not the number of objects in RAM, see relevant code where cache initialization happens