scaling prometheus, how to update the value of modulus dynamically?

[guanglinlv]

hi folks,

i'm trying to let prometheus could be scaling, I have 2 slave Prometheis, each scraping a subset of the targets. A master federates the aggregates produced by the slaves, and then the master aggregates them up to the job level. the scrape config like this:

scrape_configs:
  - job_name: overwritten-default
    # Add usual service discovery here, such as target_groups
  consul_sd_configs:
    server: <consul_address>

  relabel_configs:
    - source_labels: [__address__]
      modulus:       2    # 2 slaves
      target_label:  __tmp_hash
      action:        hashmod
    - source_labels: [__tmp_hash]
      regex:         ^1$  # This is the 2nd slave
      action:        keep
    - source_labels: ['__meta_consul_service_id']
      regex:         '^prometheus_.*'
      action:        'kee

but, the value of modulus is static, i have to update the configuration for each exist prometheus slave when scaling out some new prometheus slaves.

is there a way to get some tags from consul dynamically, as the value of modulus for each relabeling process?

thanks.

[brian-brazil]

On 22 February 2016 at 08:31, Lv,guanglin notifications@github.com wrote:

hi folks,

i'm trying to let prometheus could be scaling, I have 2 slave Prometheis,
each scraping a subset of the targets. A master federates the aggregates
produced by the slaves, and then the master aggregates them up to the job
level. the scrape config like this:

scrape_configs:

• job_name: overwritten-default

  Add usual service discovery here, such as target_groups

  consul_sd_configs:
  server: <consul_address>

  relabel_configs:

- source_labels: [__address__]
  modulus:       2    # 2 slaves
  target_label:  __tmp_hash
  action:        hashmod
- source_labels: [__tmp_hash]
  regex:         ^1$  # This is the 2nd slave
  action:        keep
- source_labels: ['__meta_consul_service_id']
  regex:         '^prometheus_.*'
  action:        'kee

but, the value of modulus is static, i have to update the configuration
for each exist prometheus slave when scaling out some new prometheus slaves.

is there a way to get some tags from consul dynamically, as the value of
modulus for each relabeling process?

There isn't. This is something that you'd typically only have to bump every
year or two (and even then that's a lot), so shouldn't be a problem in
practice. Having slaves should only be required when you have many
thousands of targets, and is not something to be done lightly.

How many samples/s are you scraping?

Brian Brazil
www.robustperception.io

[brian-brazil]

I'm presuming this is answered, if not let us know.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.