Failed to watch *v1.Node: unknown (get nodes)

[Cloudzp]

Proposal

Use case. Why is this important?

“Nice to have” is not a good use case. :)

Bug Report

What did you do?
Complete collection of kubernetes monitoring information using prometheus
What did you expect to see?
Successfully collected available information
What did you see instead? Under which circumstances?
The startup failed, and I got the error in the log, I don't know where the problem is, trouble to help me answer, thank you very much

Environment

• System information:

• Prometheus version:

• Prometheus configuration file:

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'codelab-monitor'
scrape_configs:
  - job_name: 'kubernetes-nodes-cadvisor'
    kubernetes_sd_configs:
    - api_server: 'https://kubernetes.default.svc:443'
      role: node
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        insecure_skip_verify: true
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

• Logs:

filename=/opt/prometheus/prometheus.yml
level=info ts=2019-01-20T15:14:49.302642839Z caller=main.go:657 msg="Completed loading of configuration file" filename=/opt/prometheus/prometheus.yml
level=info ts=2019-01-20T15:14:49.302671952Z caller=main.go:530 msg="Server is ready to receive web requests."
level=error ts=2019-01-20T15:14:49.315922104Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:50.319964125Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:51.326501911Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:52.329868125Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:53.333643713Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:54.337369469Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:55.340841373Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:56.34426644Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:57.351593055Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:372: Failed to watch *v1.Node: unknown (get nodes)"
level=error ts=2019-01-20T15:14:58.363697008Z caller=glog.go:94 component=k8s_client_runtime func=ErrorDepth 

[gouthamve]

Could you also let us know the version of Kubernetes you are running?

[Cloudzp]

prometheus 需要watch node资源的变化，需要在clusterroles中给prometheus配置watch的权限

- apiGroups: ['']
  resources: [node]
  verbs: [get, list, watch]