support more than one prometheus server

[glommer]

We have right now in master some metrics that are potentially very expensive to be exported, as they are O(#CFs) in size.

The idea is to export them through a secondary prometheus server, running alongside the main prometheus server. Customer interested in those metrics could enable the collection of the secondary server.

[glommer]

I just came across a use case that could greatly benefit from this.

One of our users, on slack, complains that we export too many metrics, which impacts their existing prometheus servers both in terms of network traffic and storage costs.

He very rightfully asks:

Can you configure the scrape period per metric? Like disk space is something that doesn't need to be scraped every 5s or even 30s.

We had buried the two servers discussions in the past because we would still want to see metrics for everything , and having some metrics disabled could be dangerous since when you need it they won't be there.

However, we could configure different metrics to scrape in different intervals and two prometheus servers could be an easy way to do that.
(another way would be to have a separate configuration file per metric, having different scrape periods, but that sounds like too much trouble to me)

[amnonh]

That was the exact idea behind multiple prometheus servers.

The solution was composed of two functionality (that strangely enough, only one got merged).

1. to allow programmer, when creating a metric register a metric on a specific Prometheus server (as opposed to the default Prometheus server).
2. All query subset of metrics based on their name (this part was merged).

If the two were merged, it would have allowed us to set a minimal set of metrics that will be included by default and set the configuration for more static values (like diskspace) or data hungry (like table specific) to have a lower scrap interval.

[glommer]

Turns out though there is a much simpler solution. Check our dev mailing list for a suggestion coming from a user. We can use collectors to select which metrics we will scrape - in the same server.

…

On Wed, Oct 31, 2018, 1:35 AM Amnon Heiman, ***@***.***> wrote: That was the exact idea behind multiple prometheus servers. The solution was composed of two functionality (that strangely enough, only one got merged). 1. to allow programmer, when creating a metric register a metric on a specific Prometheus server (as opposed to the default Prometheus server). 2. All query subset of metrics based on their name (this part was merged). If the two were merged, it would have allowed us to set a minimal set of metrics that will be included by default and set the configuration for more static values (like diskspace) or data hungry (like table specific) to have a lower scrap interval. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2556 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUNve0WXleR49wSOjEIN9HQ4BPunT4_ks5uqTaagaJpZM4ORVYF> .

[tzach]

We can use collectors to select which metrics we will scrape - in the same
server.

Was this ever enabled?