After restart Prometheus, it try to dump all tsdb data to memory - prometheus-2.0

[tinytub]

What did you do?
Try to benchmark prometheus 2.0.0-alpha-3
and set prometheus scrape-interval to 1s , then scrape 6 test host and they have tatal 50w samples .
After one hour, prometheus still scrape, but memory use is about 12g and still raise.
I killed prometheus and try to restart it, the main process is start,
log stop at "INFO[0000] Starting tsdb source="main.go:210""
it looks like prometheus try to dump all tsdb data to memory, after 30 minutes, prometheus memory res usage is about 33g, and my prometheus data dir is same size.

finally after another 10 minutes I got "tsdb started" log ,but the memory usage not go down

Environment

• System information:

Linux 2.6.32-642.15.1.el6.x86_64 x86_64

• Prometheus version:

prometheus, version 2.0.0-alpha.3 (branch: master, revision: 70f96b0)
build user: root@5630fb1ab539
build date: 20170622-10:04:46
go version: go1.8.3

[fabxc]

they have tatal 50w samples

What does that mean?

What are you scraping from these host in general. By default we keep the last 4h in memory. At 1s scrape intervals that naturally has implications for the memory consumed in total.

At this interval you should probably set --storage.tsdb.min-block-duration to a value like 5m. I would also recommend building from the dev-2.0 branch's HEAD as some issues were fixed in the meantime.

[tinytub]

total 500000 metrics,
I edit the example in https://github.com/prometheus/client_golang
Then I can get about 80000 metrics from every host.

Thanks for the help, I'll try it

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.