Prometheus Consuming the Entire Server Memory

[Neru007]

What did you do?
Prometheus is using too much memory and process is getting killed with out of the Memory exception. Infact kernel is killing the entire process.

Environment
Red Hat Enterprise Linux Server release 7.3

• System information:
  Linux 3.10.0-514.el7.x86_64 x86_64

• Prometheus version:
  prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98)
  build user: root@615b82cb36b6
  build date: 20171108-07:11:59
  go version: go1.9.2

• Logs:

Feb 14 14:40:24 P-RH-PRM-Prometheus kernel: Out of memory: Kill process 4431 (prometheus) score 976 or sacrifice child
Feb 14 14:40:24 P-RH-PRM-Prometheus kernel: Killed process 4431 (prometheus) total-vm:31528704kB, anon-rss:15850360kB, file-rss:0kB, shmem-rss:0kB

Also I've read on similar issues that there is a flag -storage.local.memory-chunks=1048576: but I don't find this in version 2.

It's a 16 GB server.

Mem:          15886         178         164           8       15543       15349
Swap:             0           0           0

I've also attached the Go Profiling for prometheus.
heap.svg.zip

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.