Kubernetes scrape not using TLS_config

[rikatz]

What did you do?
Configured Prometheus to scrape metrics from Kubernetes. But Prometheus keeps returning errors as it cannot auth against the API Server

The following is the config used:

- job_name: 'kubernetes-nodes'
  tls_config:
    cert_file: '/srv/conftemp/prometheus-k8s.pem'
    key_file: '/srv/conftemp/prometheus-k8s.key'
    ca_file: '/srv/prometheus2/etcdmon/ca.crt'
    insecure_skip_verify: true
  scheme: https
  kubernetes_sd_configs:
  - role: node
    api_server: https://master.lab.local
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics

But while using this, the following error is returned:

level=error ts=2018-02-16T20:01:17.097515543Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"

Note that it seems Prometheus is not using tls_config, and going directly to the API Server (User: system:anonymous)

What did you expect to see?
Scraping from Kubernetes endpoints

What did you see instead? Under which circumstances?

level=error ts=2018-02-16T20:01:17.097515543Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"

Environment

• System information:

Linux 3.10.0-514.26.2.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d82a045d103ea7f3c89a91fba4a93e6367a)
  build user:       root@6e784304d3ff
  build date:       20180119-12:01:23
  go version:       go1.9.2

• Prometheus configuration file:

scrape_configs:
- job_name: 'kubernetes-nodes'
  tls_config:
    cert_file: '/srv/conftemp/prometheus-k8s.pem'
    key_file: '/srv/conftemp/prometheus-k8s.key'
    ca_file: '/srv/prometheus2/etcdmon/ca.crt'
    insecure_skip_verify: true
  scheme: https
  kubernetes_sd_configs:
  - role: node
    api_server: https://master.lab.local
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-cadvisor'
  scheme: https
  tls_config:
    cert_file: '/srv/conftemp/prometheus-k8s.pem123'
    key_file: '/srv/conftemp/prometheus-k8s.key'
    ca_file: '/srv/prometheus2/etcdmon/ca.crt'
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: node
    api_server: https://master.lab.local
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

• Logs:

level=error ts=2018-02-16T20:01:12.089501228Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"
level=error ts=2018-02-16T20:01:12.089616635Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"
level=error ts=2018-02-16T20:01:13.091373664Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"
level=error ts=2018-02-16T20:01:13.091375642Z caller=main.go:221 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: User \"system:anonymous\" cannot list nodes at the cluster scope. (get nodes)"

[brian-brazil]

You want the tls_config of kubernetes_sd_configs.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.