The memory Prometheus uses is going up and up, crashed my machine

[gogoods]

prometheus, version 1.7.1

I run Prometheus with default configuration, but its memory is going up and up, my machine's mem is 32G, and Prometheus crashed my machine.

Even though I configured Prometheus with specified mem-chunks, like this:

nohup ./prometheus -config.file=prometheus.yml \
-storage.local.retention=2160h \
-storage.local.memory-chunks=5242880 \
-storage.local.max-chunks-to-persist=3097152 \
-storage.local.num-fingerprint-mutexes=10240 \
-storage.local.series-file-shrink-ratio=0.2 \
-storage.local.checkpoint-interval=5m \
&> prometheus.log &

but the memory problem still happens.

Why?

[gogoods]

By the way, after my machine crashed, I restarted my machine and Prometheus, then the crashrecovery started, what drives me crazy is that crashRecovery spent quite a long time, more than one hour, and still not finished

time="2017-08-07T17:47:01+08:00" level=info msg="37900000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37910000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37920000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37930000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37940000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37950000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37960000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37970000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:01+08:00" level=info msg="37980000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:02+08:00" level=info msg="37990000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:02+08:00" level=info msg="38000000 metrics queued for indexing." source="crashrecovery.go:534" 
time="2017-08-07T17:47:02+08:00" level=info msg="38010000 metrics queued for indexing." source="crashrecovery.go:534"

WTF!!!

the data path is 300+MB.

Help me

[fabxc]

Please direct usage questions to the Prometheus users mail list.

[beorn7]

You have a lot of metrics in your database. Depending on how many are active at the same time, 32GiB of RAM is probably way too small.

What you describe is the expected behavior. You need to tweak flags and/or your setup and/or move to a bigger machine. Discussing those things makes more sense if it happens on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your questions, and the whole community can benefit from the answers provided.

[gogoods]

@beorn7
thx to your attention, but why the memory exceeds what configured too much,

See, I just use the default configuration.

[beorn7]

As said, please bring your questions to the prometheus-users mailing list.

[gogoods]

@beorn7 okok,
can I ask two questions else?

[gogoods]

@beorn7

Q1: Does Prometheus support dynamic granularity storage like rrdtool?

For example:
If I set scrape-interval=30s,
in last 1st month, data's granularity is 30s;
in last 2nd month, data's granularity is 60s;
in last 3rd month, data's granularity is 90s;
...

just like the granularity can be adjusted dynamicly by the peroid in storage.

---

If not, will Prometheus support this function in the near future?

[gogoods]

@beorn7

Q2: I want to store nearly 2000 machines' metrics and there are 70 metrics of each machine, so what kind of configuration or performance of my Prometheus server needs at least?

Looking forward to your suggestion.

Best regard!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.