Recommended settings for running a machine with 2GB ram

[svenmueller]

Hi,

We are using a machine with 2CPUs/2GB RAM (ubuntu 14.04) where prometheus and alertmanager are running as Docker containers.

Currently the prometheus application runs OOM at least once day. The Prometheus server instance scrapes < 30 targets (every 15s) to collect the node machine metrics exported by node-exporter on the targets.

Which settings would you recommend to run this kind of setup? Can the settings be tuned to get a stable Prometheus or is the machine simply to small to handle this scenario?

Thx,
Sven

Current cmd line flags for docker container

-config.file=/etc/prometheus/prometheus.yml
-storage.local.path=/prometheus
-web.console.libraries=/etc/prometheus/console_libraries
-web.console.templates=/etc/prometheus/consoles
-web.external-url=https://xyz
-alertmanager.url=http://alertmanager:9093

Stackstrace

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xfca050, 0x16)
    /usr/local/go/src/runtime/panic.go:547 +0x90
runtime.sysMap(0xc8864e0000, 0x100000, 0x0, 0x1528e98)
    /usr/local/go/src/runtime/mem_linux.go:206 +0x9b
runtime.(*mheap).sysAlloc(0x150e380, 0x100000, 0x1061ea670)
    /usr/local/go/src/runtime/malloc.go:429 +0x191
runtime.(*mheap).grow(0x150e380, 0x28, 0x0)
    /usr/local/go/src/runtime/mheap.go:651 +0x63
runtime.(*mheap).allocSpanLocked(0x150e380, 0x28, 0xc81ccd9400)
    /usr/local/go/src/runtime/mheap.go:553 +0x4f6
runtime.(*mheap).alloc_m(0x150e380, 0x28, 0x100000000, 0x150f8f0)
    /usr/local/go/src/runtime/mheap.go:437 +0x119
runtime.(*mheap).alloc.func1()

Runtime Information

Uptime: 2016-07-20 10:44:09.210337115 +0000 UTC

Build Information

Version: 0.20.0
Revision: aeab25c
Branch: master
BuildUser: root@77050118f904
BuildDate: 20160616-08:38:14
GoVersion: go1.6.2

[beorn7]

See https://prometheus.io/docs/operating/storage/

I'd start with -storage.local.memory-chunks=500000. As long as you don't have more than ~200k time series, you should be fine with that (check metric prometheus_local_storage_memory_series).

[svenmueller]

Thx for the input!

Here is s screenshot from the prometheus overview console;

graph for prometheus_local_storage_memory_series (last 2h), current value: 13341

[beorn7]

Yeah, light load. You are only running out of memory because Prometheus tries to utilize ~4GiB by default. With the setting above, you should be all good.

[svenmueller]

Ok, that sounds reasonable to me. Will try it out. Thx for the support! 👍

[svenmueller]

I did set the value storage.local.memory-chunks to "500000" but still all the memory gets eaten up slowly but steadily on the Prometheus server instance.

[beorn7]

With the settings, I expect Prometheus to fully utilize your memory.
In general, you want to use all your memory, but not more. ;)

If you still run OOM, you can tweak the flag even lower. More context and tweaks are described at https://prometheus.io/docs/operating/storage/

Ideally, Prometheus would auto-tune memory usage, but that's a non-trivial problem, see #455 .

[svenmueller]

In the meantime i was trying storage.local.memory-chunks with value "400000" but still i'm seeing Prometheus running into OOMs

[beorn7]

Try lower values until you don't run into OOMs anymore.

You can also perform heap profiling to find out where the memory is gone, start with go tool pprof http://localhost:6060/debug/pprof/heap - details as https://blog.golang.org/profiling-go-programs , https://golang.org/pkg/net/http/pprof/

[philicious]

@svenmueller I'm struggling with a similar issue. if you found a storage.local.memory-chunks value thats good for you, I'd appreciate if you let me know. I'm down to 393216 which is maxing out at 2.2GB as it seems. With short bursts to 2.6. (For this particular environment I'd like it to be at ~1GB. dont know if thats even possible. Usually I just give it as much RAM as needed, which I cant here)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.