How to configure kubernetes_sd_configs to discover services in a different cluster

[ahmed282828]

I created a job in prometheus with kubernetes_sd_configs to discover services automatically without the need to declare them manually(Cadvisor, nodeexporter, KubeStateMetrics). i am using helm chart for prometheus. i tried this job and it can only discover the services in the cluster itself(same cluster with prometheus server installed) but not the other clusters

this is the job that i created. in prometheus the $1 takes the names of the old cluster and the error displayed is server returned HTTP status 401 Unauthorized

-job_name: 'kubernetes-nodes-cadvisor-aks-NEWUATBACK'
scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

  insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
  - role: node

relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: ks-ba4cf716.hcp.northeurope.azmk8s.io:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

anybody has an idea about this ? i really need your help guys am really struggling with this and i didnt find a solution

[simonpasquier]

From the documentation of kubernetes_sd_config:

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

To discover targets from another Kubernetes cluster, you need to pass the api_server URL as well as TLS certificate or bearer token files for Prometheus to authenticate.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[ahmed282828]

thank you a lot simonpasquier! am gonna try this!