Prometheus Memory Issues

[lswith]

Prometheus has been running in production for about 4 months now. It has been working great however trying to configure the memory usage is still a problem.
Currently our server has 64Gb of RAM available. We have set our memory-chunks to 10485760 which is well below the 3x threshold suggested.

Currently the memory usage is something along these lines:
https://snapshot.raintank.io/dashboard/snapshot/GD6AZ2rTkuMttT2JjOWezEkz9ABoh53c

Basically the memory usage hovers around 30-50Gb then somehow jumps up to max RAM and CPU goes up.

I thought it might be querying but it seems that the query rate hasn't changed. The amount of Pins and Unpins however does jump.

Environment

• System information:

Linux 4.4.0-38-generic x86_64

• Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@a8af9200f95d
  build date:       20170210-14:41:22
  go version:       go1.7.5

• Prometheus Service Configuration

prometheus -config.file=/etc/prometheus/prometheus.yaml -storage.local.memory-chunks=10485760 -storage.local.retention=8760h0m0s -storage.local.series-file-shrink-ratio=0.3 -storage.local.max-chunks-to-persist=5242880

[beorn7]

Your flags look good. That's pretty much the configuration I'd use myself. Not that the suggested 3x factor is a lower bound. (We have some servers at SoundCloud with a lot of targets, series, and very involved Kubernetes Service Discovery and relabeling, where we need to apply a 10x factor!)

Sadly, memory usage in a Go program is really hard to predict/control. There are a million things that require more or less memory at different times.

But I'll give you three lights at the end of the tunnel:

1. (Shameless plug!) I'll give a talk about the memory (and other) tuning at CloudNativeCon. AFAIK all those talks will be recorded and made publicly available.
2. The next minor release (1.6) will give you the option to set a target heap size, which will be much more convenient because the server will react to sudden memory demands by changing the memory chunks dynamically.
3. The next major release (2.x, ETA unknown) will feature a completely re-written storage layer where memory management will be a non-issue.

If you have more questions, I recommend to discuss them on the prometheus-users mailing list.

[lswith]

that's awesome to hear! Looking forward to the next release.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.