request - support scrape_interval override in relabel_configs

[cameronbraid]

In kubernetes/openshift, I would like to be able to use annotations on my pods that define the scrape_interval

i.e. I would like to do something like this in the relabel config :

 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_interval]
    action: replace
    regex: (.+)
    target_label: __scrape_interval__
    replacement: $1

Then I could use annotations on pods/deployment configs such as :

 annotations:
   prometheus.io/port: '8081'
   prometheus.io/path: '/prometheus'
   prometheus.io/scrape_interval: '5m'

[brian-brazil]

This has been discussed before and will not be supported. In general you should almost always have the same scrape interval for an entire Prometheus.

[brancz]

We are tackling something in that direction with the prometheus-operator. Currently this only works on Service objects with the respective ServiceMonitor. We are planning to expand this to Pods though.

The idea is that teams or a company likely have some common practice on metrics and that can be expressed with one ServiceMonitor or in the future also a PodMonitor. Feedback is appreciated! 🙂

[zegl]

@brian-brazil I'm interested in knowing the reasoning behind this decision, and I'd like you to reconsider this it.

Some exporters use a lot of resources (examples: prometheus-kafka-consumer-group-exporter, jmx_exporter for Cassandra), and often you're willing to reduce the scrape interval for services to keep the resource utilization down.

In the cloudwatch_exporter the scrape interval is directly linked to the cost of running the exporter. In the recommended setup data newer than 10 minutes is ignored anyway, so scraping it more often than lets say every 5 minutes is a complete waste of money.

A workaround is obviously to setup a separate scraping job that scrapes all pods in that job less frequently, but it's annoying to do it if you have lots of applications with different scrape intervals.

[lsowen]

@brian-brazil I have the same use case. I want to decrease the scrape_interval for the cloudwatch_exporter to something like every 5m, but keep the rest of the on cluster intervals to 60s.

It looks like a custom scrape_interval is supported for scrape_config, just not exposed as a relabel_configs target.

Edit:

Looks like there is some ongoing discussion about this in #4561

[brian-brazil]

For slow exporters like cloudwatch and certain JMX uses the recommendation is to have a separate scrape config (don't go over 2m). These cases are (thankfully) fairly rare though. This does not require relabelling.

so scraping it more often than lets say every 5 minutes is a complete waste of money.

Delay and precision are different things, so this is not true.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.