Empty node metrics with kubernetes v 1.11.0

[nikhilkhanna]

When using the metrics server with the latest version of kubernetes the command kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes outputs

{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[]}

where there is an emtpy list for the node metrics even though this is a on a cluster I know has 4 nodes as kubectl get nodes prints out

NAME               STATUS    ROLES     AGE       VERSION
ip-172-31-17-53    Ready     <none>    21m       v1.11.0
ip-172-31-23-195   Ready     <none>    48s       v1.11.0
ip-172-31-24-3     Ready     master    25m       v1.11.0
ip-172-31-27-90    Ready     <none>    21m       v1.11.0
ip-172-31-31-206   Ready     <none>    21m       v1.11.0

[zreigz]

I have installed the metrics-server on k8s v 1.11.0 and it seems like problem with service account:

E0704 17:45:41.232723       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0704 17:45:41.233424       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0704 17:46:05.006896       1 summary.go:97] error while getting metrics summary from Kubelet lukasz(192.168.0.100:10255): Get http://192.168.0.100:10255/stats/summary/: dial tcp 192.168.0.100:10255: getsockopt: connection refused

[wangsenyuan]

I have a similar problem, with kubernetes v1.11.0;
error while getting metrics summary from Kubelet k8s1(192.168.5.110:10255): Get http://192.168.5.110:10255/stats/summary/: dial tcp 192.168.5.110:10255: getsockopt: connection refused

I noticed that, the kubelet doesn't listen on the port 10255 on every nodes. netstat -tulpn | grep 10255 show nothing.

[nasuku]

I faced the same problem too.

Solution that worked for me is to pass the following command line option:

• --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

i.e
$ kubectl -n kube-system edit deploy metrics-server

look in section spec/template/spec/command for something like:
- --source=kubernetes
and replace that line with
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

[amitvashist7]

Yes the above-mentioned workaround work for me too!!!

Before the changes errors:

E0710 20:33:05.019675 1 summary.go:97] error while getting metrics summary from Kubelet worker01(172.31.0.11:10255): Get http://172.31.0.11:10255/stats/summary/: dial tcp 172.31.0.11:10255: getsockopt: connection refused
E0710 20:33:05.021822 1 summary.go:97] error while getting metrics summary from Kubelet worker02(172.31.0.12:10255): Get http://172.31.0.12:10255/stats/summary/: dial tcp 172.31.0.12:10255: getsockopt: connection refused
root@kube-master:~/kubernetes-project/metrics-server#``

After replacing the source value to :
From: - --source=kubernetes.summary_api:''
To : --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

Getting the below expected output

root@kube-master:~/kubernetes-project/metrics-server# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
kube-master 342m 17% 1289Mi 67%
worker01 131m 6% 599Mi 67%
worker02 134m 6% 624Mi 70%

root@kube-master:~/kubernetes-project/metrics-server# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
echoheaders-7ftsg 0m 15Mi
helloworld-v1-deployment-54db457d99-djpff 0m 57Mi
helloworld-v2-deployment-7c74bd58bf-tqzn5 0m 57Mi
nginx-ingress-controller-thxz5 1m 53Mi
root@kube-master:~/kubernetes-project/metrics-server#

[DirectXMan12]

Yes, kubernetes recently started disabling the read-only port by default. I've got a PR in that changes metrics-server to use the secure port by default (part of #65).

[adventurer2008]

When I did the folowing step:

in section spec/template/spec/command for something like:

• --source=kubernetes
  and replace that line with
• --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

I got the erros as below:

namespace: kube-system
[jboss@lm2app01 1.8+]$ kubectl -n kube-system logs $(kubectl get pods --namespace=kube-system -l k8s-app=metrics-server -o name)
I0718 06:55:55.400061 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
I0718 06:55:55.400208 1 heapster.go:72] Metrics Server version v0.2.1
I0718 06:55:55.400979 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version
I0718 06:55:55.401003 1 configs.go:62] Using kubelet port 10250
I0718 06:55:55.401789 1 heapster.go:128] Starting with Metric Sink
I0718 06:55:55.878316 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0718 06:55:56.966204 1 heapster.go:101] Starting Heapster API server...
[restful] 2018/07/18 06:55:56 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/07/18 06:55:56 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0718 06:55:56.968140 1 serve.go:85] Serving securely on 0.0.0.0:443
E0718 06:56:05.028363 1 summary.go:97] error while getting metrics summary from Kubelet lm2app01(10.100.40.187:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.031819 1 summary.go:97] error while getting metrics summary from Kubelet lm3app52(10.100.40.146:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.032233 1 summary.go:97] error while getting metrics summary from Kubelet lm3app53(10.100.40.178:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.036292 1 summary.go:97] error while getting metrics summary from Kubelet lm2doc02(10.100.40.191:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"

Could anyone help me ?

By changing RBAC authorization, it's fixed.

[DirectXMan12]

yeah, it looks like you're missing the RBAC to let metrics server access the nodes/stats subresource. There was a change around that recently, so just make sure the copy of the RBAC info is up to date. If it is up-to-date, please point me at the RBAC definitions that you used so we can make sure they're correct.

[LucasSales]

I was having this same problem in the version v0.3.0 do metrics "error while getting metrics summary from Kubelet k8s1(:10255): Get http://:10255/stats/summary/: dial tcp :10255: getsockopt: connection refused". To solve, i use the command $ kubectl -n kube-system edit deploy metrics-server and on session spec/template/spec/containers, i add

• command:
  - /metrics-server
  - --kubelet-insecure-tls.

This procedure solve my problem

[leandroaur]

Unfortunately I am still experiencing this issue for arm v7 (raspberry 4B) even though editing metrics server on both methods:

        command:
        - /metrics-server
        - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

Or using

         command:
         - /metrics-server
         - --kubelet-insecure-tls
         - --kubelet-preferred-address-types=InternalIP

I still get no target for my cpu usage

kubectl get hpa
NAME        REFERENCE              TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
wordpress   Deployment/wordpress   <unknown>/50%   1         10        3          16m