prometheus can't scrap kubelet target "Error: serevr returned HTTP status 401 Unauhtorized"

[Mnbill]

Bug Report

What did you do?
I installed prometheus to monitor my kubernetes cluster.

What did you expect to see?
prometheus scrap metrics from my kubernetes cluster,
In prometheus dashbored target
all targets on UP status

What did you see instead? Under which circumstances?
My prometheus can't scrap metrics
In prometheus dashbored target
my kubelete target is Down with Error "serevr returned HTTP status 401 Unauhtorized"
and kube-dns is donws with Error "Get http://10.13.2.39:10054/metrics: dial tcp 10.13.2.39:10054: connect: connection refused"

PS my apiserver and kube-state are UP

Environment

• System information:

  OS: ubuntu 16.04 xenial
  Kernel: Linux 4.15.0-1036-azure

• Kubernetes:

     Azure acs-engine 
     Client Version: v1.10.3
     Server Version: v1.10.1
  

• Docker: v1.13.1

• Prometheus version: v2.3.1

• Prometheus configuration file:

scrape_configs:


  honor_labels: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: exporter-kube-state
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: kube-state
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: kube-state-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: kube-state-metrics
    action: replace
- job_name: monitoring/kube-prometheus-exporter-kubelets/0
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace


  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: exporter-kube-controller-manager
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: kube-controller-manager
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
- job_name: monitoring/kube-prometheus-exporter-kube-dns/0
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: exporter-kube-dns
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: kube-dns
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics-dnsmasq
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics-dnsmasq
    action: replace

• Logs:

*prometheus pod logs*
level=info ts=2019-02-05T11:16:19.001422149Z caller=kubernetes.go:188 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-02-05T11:16:19.002529055Z caller=kubernetes.go:188 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-02-05T11:16:19.003831662Z caller=kubernetes.go:188 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
level=error ts=2019-02-05T11:16:19.004073863Z caller=endpoints.go:130 component="discovery manager notify" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"

Kubelet logs
systemctl status kubelet

logging error output: "Unauthorized"
[[Prometheus/2.3.1] 10.15.5.3:56254]

kube-dns pod logs

I0204 22:35:41.179355       1 logs.go:41] skydns: failure to forward request "read udp 10.13.2.67:46261->168.63.129.16:53: i/o timeout"
I0205 10:50:55.002720       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 10:50:55.017169       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 10:50:55.040121       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 10:50:55.079320       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:10:29.619533       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:10:29.638260       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:10:29.645160       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:10:29.676569       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:14:43.704733       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:14:43.727123       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:14:43.750408       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0205 11:14:43.790782       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.

[simonpasquier]

Thanks for your report. It looks as if this is actually a question about usage and not development.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.