Prometheus high memory usage,How should i optimize?

[barnettZQG]

Now,my prometheus server manager the exporter of number more than 160,including node-exporter,cAdvisor and mongodb-exporter. scrape them every 5 seconds.
I saw Prometheus resource usage about as follows:
CPU:100% to 1000%（The machine has 10 CPU, 40 CORE）
MEM:90G
The above data is normal?How to optimize them?Because of my monitoring needs to rise to more than 1000 exporter

[fabxc]

You find documentation about tweaking memory usage here:
https://prometheus.io/docs/operating/storage/#memory-usage

90G seems rather high, but cAdvisor can accumulate quite a lot of metrics,
so not impossible.

On Wed, Aug 10, 2016 at 4:35 AM barnettZQG notifications@github.com wrote:

Now,my prometheus server manager the exporter of number more than
160,including node-exporter,cAdvisor and mongodb-exporter. scrape them
every 5 seconds.
I saw Prometheus resource usage about as follows:
CPU:100% to 1000%（The machine has 10 CPU, 40 CORE）
MEM:90G
The above data is normal?How to optimize them?Because of my monitoring
needs to rise to more than 1000 exporter

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1881, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuA8lXxzL0A0ao34m1uDRYrGwYasX_gks5qeTkFgaJpZM4Jgtek
.

[barnettZQG]

The metrics number of cAdvisor less than node-exporter.

[juliusv]

@barnettZQG Some information about the total number of memory series, ingested samples per second, in-memory chunks, and received queries would be interesting. Could you execute the following queries and tell us the results:

• rate(prometheus_local_storage_ingested_samples_total[5m])
• prometheus_local_storage_memory_series
• prometheus_local_storage_memory_chunks
• rate(http_request_duration_microseconds_count{handler=~"query|query_range"}[5m])

[barnettZQG]

@juliusv
rate(prometheus_local_storage_ingested_samples_total[5m]):100000（avg）
prometheus_local_storage_memory_series: 754439
prometheus_local_storage_memory_chunks:1144916
rate(http_request_duration_microseconds_count{handler=~"query|query_range"}[5m]):query and query_range number is very few

[barnettZQG]

@juliusv
The current number of chunks waiting for persistence add more than 1000 per a second,so that more than the local_storage_chunks_to_persist after a period of time.
local_storage_chunks_to_persist:10485760
local_storage_memory_chunks:20971520

I thank is there some problem with IO?

[dobesv]

This link https://prometheus.io/docs/operating/storage/#memory-usage doesn't seem to lead to any information about prometheus memory usage (any more). It just discusses on disk usage.

[juliusv]

@dobesv Good point, the three "Troubleshooting" FAQ items are about Prometheus 1.x, so we should either remove them or clarify that they are about 1.x and qualify the link to point at the versioned docs at https://prometheus.io/docs/prometheus/1.8/storage/#memory-usage

[juliusv]

I filed a pull request to qualify the version: prometheus/docs#998

[dobesv]

Heh, unfortunately I'm using prometheus 2.0.0 and running out of memory. Where should I look for guidance on understanding and managing memory consumption ?

[juliusv]

I'm not sure that there's already a doc about 2.x memory consumption anywhere.

@fabxc or @gouthamve can probably give the best explanation, but according to my understanding:

• The TSDB head block holds the last 2-3h of all series directly in memory (in normal Go datastructures). The usage there is proportional to the number of concurrently active series you have, and also to the sample ingestion rate. Not sure what the memory bytes/sample or bytes/series roughly is for Prometheus 2.x head blocks.
• Older data blocks are mmaped from disk, so the OS has more control here over what data to page in/out based on memory pressure, so there's more elasticity there.
• Queries, ingestion, remote write, and other features also use up memory, but I also don't have any good rule-of-thumb guidelines for estimation around those. Especially query memory usage will depend a lot on the type of query, what data it selects, etc.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.