Prometheus federation

[vivekbny]

###Hi All,

We are trying to cluster the prometheus instances. We are creating two prometheus instances. if one goes down(can be any reason), we should be able to retreive the data from the other instance.

Below is our yml file for 1st prometheus instance.(r0106sn0v)
#my global config
global:
scrape_interval: 15s #By default, scrape targets every 15 seconds.
evaluation_interval: 15s #By default, scrape targets every 15 seconds.
#scrape_timeout is set to the global default (10s).

Attach these labels to any time series or alerts when communicating with
#external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'prometheus.Prometheus-PRD-Env'

#Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:

• "alert.rules"
  #- "first.rules"
  #- "second.rules"

#A scrape configuration containing exactly one endpoint to scrape:
#Here it's prometheus.Prometheus itself.
#DEV 1 Cluster
scrape_configs:
#The job name is added as a label job=<job_name> to any timeseries scraped from this config.

• job_name: 'Cluster01'

  #Override the global default and scrape targets from this job every 5 seconds.
  scrape_interval: 60s

  #metrics_path defaults to '/metrics'
  #scheme defaults to 'http'.

  static_configs:
  - targets: ['r01055n0v.bnymellon.net:9100']

• job_name: 'federate'
  scrape_interval: 15s
  honor_labels: true
  metrics_path: '/federate'

  params:
  match[]: [job=="Cluster01"]

  static_configs:
  - targets: ['r01071n0v.bnymellon.net:9090']

for 2nd prometheus instance(r01071n0v)
#my global config
global:
scrape_interval: 15s #By default, scrape targets every 15 seconds.
evaluation_interval: 15s #By default, scrape targets every 15 seconds.
#scrape_timeout is set to the global default (10s).

#Attach these labels to any time series or alerts when communicating with
#external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'prometheus.Prometheus-PRD-Env'

#Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:

• "alert.rules"
  #- "first.rules"
  #- "second.rules"

#A scrape configuration containing exactly one endpoint to scrape:
#Here it's prometheus.Prometheus itself.
#DEV 1 Cluster
scrape_configs:
#The job name is added as a label job=<job_name> to any timeseries scraped from this config.

• job_name: 'Cluster02'

  #Override the global default and scrape targets from this job every 5 seconds.
  scrape_interval: 60s

  #metrics_path defaults to '/metrics'
  #scheme defaults to 'http'.

  static_configs:
  - targets: ['r01055n0v.bnymellon.net:9100']

• job_name: 'federate'
  scrape_interval: 15s
  honor_labels: true
  metrics_path: '/federate'

  params:
  match[]: [job=="Cluster02"]

  static_configs:
  - targets: ['r0106sn0v.bnymellon.net:9090']

When we try to access the federate URL; we are getting below error
parse error at char 4: could not parse remaining input "=="Cluster01""...

Is the above method is right way for federation??

Our current architecture is as below:
Exporters-> Promethues -> grafana

we are trying to setup the promethues in a way that if one node goes down the data should be still seen at grafana coming from the other node to avoid a single point of failure.

[beorn7]

Two mistakes here:

First, to run two Prometheus servers for redundancy and (manual) failover, run them both in parallel with the same config, i.e. both will scrape all targets directly, rather than having one federate from the other.

Second, the match[] parameter must be an instant vector selector as per https://prometheus.io/docs/operating/federation/#configuring-federation . [job=="Cluster02"] is not an instant vector selector. You probably want {job="Cluster02"}.

[gouthamve]

@brian-brazil This can be closed as it is a usage question.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.