How about the Prometheus's performance?

[guxia]

Anybody tell me how about the Prometheus's performance when it is applied to an existing project of java?As we know Prometheus can to monitor itself,is it also effect the Prometheus's performance？And how?In addition，will it effect Prometheus monitor to itself when it is applied to other project?

[fabxc]

The impact of self-monitoring is negligible since Prometheus is built to
monitor hundreds to thousands of targets. Itself is just a single one among
those.
We need some more information on what you want to monitor (how many
instances of your application, how many metrics each). The language the
application you want to monitor was written in is of no practical
performance concern.

On Fri, Apr 22, 2016 at 5:21 AM guxia notifications@github.com wrote:

In addition，will it effect Prometheus monitor to itself when it is applied
to other project?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1573 (comment)

[guxia]

fabxc，thank you answer for me.As you say 'The impact of self-monitoring is negligible',so it will more or less affect performance.And i want to know how is it produced?What is the principle?Is there any document about this?

[brian-brazil]

The current record is 800k samples/s, which is enough to monitor over 10k node exporters at a 10s interval. This presentation has more information that should help answer your questions: https://vimeo.com/173704322

[LuboVarga]

I am not able to find out how much busy is mine instance of prometheus.

Am I correct, that to find out, if querying my prometheus instance is going fast enough, I can use query like this:

rate(http_request_duration_microseconds_count{handler=~"query|metrics"}[1m])

How can I find out how much samples/s is my instance processing right now?

Is there possibility to estimate capacity of my current instance by some query?

[beorn7]

Queries vary wildly in cost. Capacity planning here really depends on the nature of your queries. It's easy to create a dashboard with a "query of death" (Prometheus will not die, but the query will run into a timeout and burn a lot of resources on its way). Usually, with creating recording rules, you can work around slow/expensive queries.

The other side is ingestion, which is way easier to reason with capacity-wise. You can find out how many samples your server is ingesting with this query:
rate(prometheus_local_storage_ingested_samples_total[2m])

[LuboVarga]

Thanks, this have helped me a bit.

One more performance related question: Is there any simple way, how to evaluate "cost" of query (something like explain analyze in postgres). Should I just use common sense to do it?

[beorn7]

We had some query analysis in the past, but that got kicked out when we killed the log spam...

I think we should have simple logging (the URL and time) on INFO log level and all the analysis on DEBUG level.

This is essentially tracked in #1315

[vjagadeesan]

Am considering Prometheus for monitoring system and application level metrics for a Java based back-office classic 3-tier system. A few questions in this regard -

1. Does Prometheus itself introduce any performance degradation on the system that it monitors? If so, is there a published ballpark on what the level of degradation could be? Can consider the recommended approach of "pull" for everything other than short-lived jobs.

2. Could there be any implications by monitoring to collect metrics at 1 sec intervals rather than a higher level of granularity like 10 or 20 secs? This is a general question for any monitoring solution and not necessarily only Prometheus.

3. Any comparisons that may have been done against commercial solutions like New Relic and the outcomes of the same?

Thank you.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.