kubernetes Failed to list *v1.Endpoints: Get https://xxxx:6443/api/v1/endpoints?limit=500&resourceVersion=0: x509: certificate signed by unknown authority"

[xiaoxi1989]

Bug Report

What did you do?
configuration in outside kubernetes cluster

- job_name: 'kubernetes-service-endpoints'

 
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://xxxx(outside cluster):6443
    tls_config:
      insecure_skip_verify: true
    bearer_token: xxxx
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

What did you expect to see?
it can get entpoints info.

What did you see instead? Under which circumstances?

level=error ts=2018-09-29T12:31:02.230105551Z caller=main.go:234 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:286: Failed to list *v1.Endpoints: Get https://xxxxxx:6443/api/v1/endpoints?limit=500&resourceVersion=0: x509: certificate signed by unknown authority"

Environment

• System information:
  Darwin 17.3.0 x86_64

• Prometheus version:
  prometheus, version 2.4.2 (branch: HEAD, revision: c305ffa)
  build user: root@dcde2b74c858
  build date: 20180921-07:27:12
  go version: go1.10.3

[xiaoxi1989]

it work:
curl -k -H "Authorization: Bearer $(cat ../kubernetes/token)" 'https://xxx:6443/api/v1/endpoints?limit=500&resourceVersion=0'

[simonpasquier]

I've tested on my local environment and it works fine with insecure_skip_verify: true. Can you paste the full configuration as displayed in the Prometheus UI (Status > Configuration menu)?

[trnl]

If cluster is with RBAC, do:

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

[vivekj11]

I was getting the same error even after mentioning insecure_skip_verify: true.
Though it worked for me in this way

kubernetes_sd_configs:
- api_server: https://<ip>:6443
  role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true

By specifying inseucre_skip_verfity as a part of api_server context

[simonpasquier]

Closing due to inactivity.