kuernetes_sd_configs kubernetes-nodes server returned HTTP status 403 Forbidden

[xikunyang]

I configure prometheus for kubernetes, but I suffer an error on kubernetes-nodes.

My prometheus.yml:

- job_name: 'kubernetes-nodes'
    scheme: https
    kubernetes_sd_configs:
    - role: node
      api_server: "https://api_server_ip:6443"
      tls_config:
        ca_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/ca.crt
      bearer_token_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/token

    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: api_server_ip:6443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics

The rbac clusterrole for prometheus:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources: ["pods", "nodes", "nodes/proxy"]
  verbs: ["get", "watch", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: default
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
roleRef:
  kind: ClusterRole
  name: prometheus
  apiGroup: rbac.authorization.k8s.io

prometheus.log doesn't show any relate error message.

I test curl , it return metrics success:

curl -H "Authorization: Bearer token" https://api_server_ip:6443/api/v1/nodes/192.168.0.112/proxy/metrics

Any help is appreciate!

[simonpasquier]

You need to pass the correct authentication parameters at the top-level too.

- job_name: 'kubernetes-nodes'
    scheme: https
   # Used by Prometheus when scraping the targets.
    tls_config:
      ca_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/ca.crt
    bearer_token_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/token
    kubernetes_sd_configs:
    - role: node
      api_server: "https://api_server_ip:6443"
     # Used by Prometheus for service discovery.
      tls_config:
        ca_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/ca.crt
      bearer_token_file: /opt/prometheus-2.2.1.linux-amd64/kubernetes/token
      ...

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.