Investigate RAM consumption during crash recovery

[beorn7]

We have received occasional reports of servers OOMing during crash recovery.

Obviously, the checkpoint has to be loaded in its' entirety, but if more is loaded from disk, it could explain the OOMing as no series maintenance or chunk eviction is running. After a quick check, I could only see chunk descs being loaded. In extreme cases, even the relatively small chunk descs might cause an OOM, so unloading chunk descs will definitely be a way to reduce RAM usage during crash recovery.

But there might be other code paths where chunks might be loaded. This has to be investigated more thoroughly.

Obviously, having #447 in place would come in handy.

@matthiasr as discussed earlier today.

[beorn7]

Random observation: A beefy Prometheus server seemed to ramp up its RAM usage during rebuilding the metrics index (xxx metrics queued for indexing).

[beorn7]

Wild guess: If LevelDB gets a lot of updates, it might run into trouble cleaning up and hogs too much RAM.

[beorn7]

I have decided to not tackle the LevelDB issues. This will be hairy at best, and it is going away in v2.0 anyway.
Evicting chunkdescs is however low hanging fruit. I'll create a PR shortly (for the 1.6 release).

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.