Configuring Prometheus for multiple Kubernetes Clusters

[NayabZehra]

What did you do?
I am trying to configure Prometheus for multiple Kubernetes Cluster and running prometheus outside the cluster.
I have used the configurations provided here https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

• Prometheus configuration file:

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'kubernetes-cadvisor'
    scrape_interval: 5s
    scheme: https
    basic_auth:
      username: xxx
      password: 'xxx'
    tls_config:
      insecure_skip_verify: true
    kubernetes_sd_configs:
    - api_server: &api_servers 'https://cluster01'
      role: node
      basic_auth:
        username: xxx
        password: 'xxx'
      tls_config:
        insecure_skip_verify: true
    - api_server: &api_servers 'https://cluster02'
      role: node
      basic_auth:
        username: xxx
        password: 'xxx'
      tls_config:
        insecure_skip_verify: true
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: cluster01 or cluster02
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

What did you expect to see?
As you can see that in the relabeling configs i need to provide the api_server address and port but there is no meta label provided for api_server. so i cant provide the api_server address to the address label. as this metrics path /api/v1/nodes/${1}/proxy/metrics/cadvisor will only work with the api server of kubernetes master, so the replacement of address should be the master ip, that will be different for all clusters. How can i achieve that?

Environment

• System information:

  Prometheus Host : Linux 4.4.0-104-generic x86_64
  Kubernetes Host : Linux 3.10.0-693.5.2.el7.x86_64 x86_64

• Prometheus version:

  v1.8.0

[brian-brazil]

You need two scrape configs to do this. Generally you'd run one Prometheus inside each cluster.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[NayabZehra]

Hi @brian-brazil
Actually I have asked this question here https://groups.google.com/forum/#!topic/prometheus-users/xjzGIuee5co it seems like i have to create more jobs but i want to use one job only for multiple clusters. so is there any way i can provide the api server to the address field, and for each cluster relabel configs will work?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.