New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty node metrics with kubernetes v 1.11.0 #77

Closed
nikhilkhanna opened this Issue Jul 4, 2018 · 8 comments

Comments

Projects
None yet
8 participants
@nikhilkhanna
Copy link

nikhilkhanna commented Jul 4, 2018

When using the metrics server with the latest version of kubernetes the command kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes outputs

{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[]}

where there is an emtpy list for the node metrics even though this is a on a cluster I know has 4 nodes as kubectl get nodes prints out

NAME               STATUS    ROLES     AGE       VERSION
ip-172-31-17-53    Ready     <none>    21m       v1.11.0
ip-172-31-23-195   Ready     <none>    48s       v1.11.0
ip-172-31-24-3     Ready     master    25m       v1.11.0
ip-172-31-27-90    Ready     <none>    21m       v1.11.0
ip-172-31-31-206   Ready     <none>    21m       v1.11.0
@zreigz

This comment has been minimized.

Copy link

zreigz commented Jul 4, 2018

I have installed the metrics-server on k8s v 1.11.0 and it seems like problem with service account:

E0704 17:45:41.232723       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0704 17:45:41.233424       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0704 17:46:05.006896       1 summary.go:97] error while getting metrics summary from Kubelet lukasz(192.168.0.100:10255): Get http://192.168.0.100:10255/stats/summary/: dial tcp 192.168.0.100:10255: getsockopt: connection refused
@wangsenyuan

This comment has been minimized.

Copy link

wangsenyuan commented Jul 5, 2018

I have a similar problem, with kubernetes v1.11.0;
error while getting metrics summary from Kubelet k8s1(192.168.5.110:10255): Get http://192.168.5.110:10255/stats/summary/: dial tcp 192.168.5.110:10255: getsockopt: connection refused

I noticed that, the kubelet doesn't listen on the port 10255 on every nodes. netstat -tulpn | grep 10255 show nothing.

@nasuku

This comment has been minimized.

Copy link

nasuku commented Jul 6, 2018

I faced the same problem too.

Solution that worked for me is to pass the following command line option:

i.e
$ kubectl -n kube-system edit deploy metrics-server

look in section spec/template/spec/command for something like:
- --source=kubernetes
and replace that line with
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

@amitvashist7

This comment has been minimized.

Copy link

amitvashist7 commented Jul 10, 2018

Yes the above-mentioned workaround work for me too!!!

Before the changes errors:

E0710 20:33:05.019675 1 summary.go:97] error while getting metrics summary from Kubelet worker01(172.31.0.11:10255): Get http://172.31.0.11:10255/stats/summary/: dial tcp 172.31.0.11:10255: getsockopt: connection refused
E0710 20:33:05.021822 1 summary.go:97] error while getting metrics summary from Kubelet worker02(172.31.0.12:10255): Get http://172.31.0.12:10255/stats/summary/: dial tcp 172.31.0.12:10255: getsockopt: connection refused
root@kube-master:~/kubernetes-project/metrics-server#``

After replacing the source value to :
From: - --source=kubernetes.summary_api:''
To : --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

Getting the below expected output

root@kube-master:~/kubernetes-project/metrics-server# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
kube-master 342m 17% 1289Mi 67%
worker01 131m 6% 599Mi 67%
worker02 134m 6% 624Mi 70%

root@kube-master:~/kubernetes-project/metrics-server# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
echoheaders-7ftsg 0m 15Mi
helloworld-v1-deployment-54db457d99-djpff 0m 57Mi
helloworld-v2-deployment-7c74bd58bf-tqzn5 0m 57Mi
nginx-ingress-controller-thxz5 1m 53Mi
root@kube-master:~/kubernetes-project/metrics-server#

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

DirectXMan12 commented Jul 13, 2018

Yes, kubernetes recently started disabling the read-only port by default. I've got a PR in that changes metrics-server to use the secure port by default (part of #65).

@adventurer2008

This comment has been minimized.

Copy link

adventurer2008 commented Jul 18, 2018

When I did the folowing step:

in section spec/template/spec/command for something like:

I got the erros as below:

namespace: kube-system
[jboss@lm2app01 1.8+]$ kubectl -n kube-system logs $(kubectl get pods --namespace=kube-system -l k8s-app=metrics-server -o name)
I0718 06:55:55.400061 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
I0718 06:55:55.400208 1 heapster.go:72] Metrics Server version v0.2.1
I0718 06:55:55.400979 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version
I0718 06:55:55.401003 1 configs.go:62] Using kubelet port 10250
I0718 06:55:55.401789 1 heapster.go:128] Starting with Metric Sink
I0718 06:55:55.878316 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0718 06:55:56.966204 1 heapster.go:101] Starting Heapster API server...
[restful] 2018/07/18 06:55:56 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/07/18 06:55:56 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0718 06:55:56.968140 1 serve.go:85] Serving securely on 0.0.0.0:443
E0718 06:56:05.028363 1 summary.go:97] error while getting metrics summary from Kubelet lm2app01(10.100.40.187:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.031819 1 summary.go:97] error while getting metrics summary from Kubelet lm3app52(10.100.40.146:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.032233 1 summary.go:97] error while getting metrics summary from Kubelet lm3app53(10.100.40.178:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"
E0718 06:56:05.036292 1 summary.go:97] error while getting metrics summary from Kubelet lm2doc02(10.100.40.191:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"

Could anyone help me ?

By changing RBAC authorization, it's fixed.

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

DirectXMan12 commented Jul 18, 2018

yeah, it looks like you're missing the RBAC to let metrics server access the nodes/stats subresource. There was a change around that recently, so just make sure the copy of the RBAC info is up to date. If it is up-to-date, please point me at the RBAC definitions that you used so we can make sure they're correct.

@LucasSales

This comment has been minimized.

Copy link

LucasSales commented Sep 6, 2018

I was having this same problem in the version v0.3.0 do metrics "error while getting metrics summary from Kubelet k8s1(:10255): Get http://:10255/stats/summary/: dial tcp :10255: getsockopt: connection refused". To solve, i use the command $ kubectl -n kube-system edit deploy metrics-server and on session spec/template/spec/containers, i add

  • command:
    - /metrics-server
    - --kubelet-insecure-tls.

This procedure solve my problem

stephenmoloney added a commit to stephenmoloney/k8s-scw-baremetal that referenced this issue Sep 8, 2018

Update RBAC settings for the metrics server
What does this commit/MR/PR do?

- Update RBAC settings for the metrics server

Why is this commit/MR/PR needed?

- Due to a bug, output below

```
error while getting metrics summary from Kubelet
amd64-node-1(10.1.244.207:10250): request failed - "403 Forbidden",
response: "Forbidden
(user=system:serviceaccount:kube-system:metrics-server, verb=get,
resource=nodes, subresource=stats)"
```

- Ref issue
`kubernetes-incubator/metrics-server#77

stephenmoloney added a commit to stephenmoloney/k8s-scw-baremetal that referenced this issue Sep 8, 2018

Update RBAC settings for the metrics server
What does this commit/MR/PR do?

- Update RBAC settings for the metrics server

Why is this commit/MR/PR needed?

- Due to a bug, output below

```
error while getting metrics summary from Kubelet
amd64-node-1(10.1.244.207:10250): request failed - "403 Forbidden",
response: "Forbidden
(user=system:serviceaccount:kube-system:metrics-server, verb=get,
resource=nodes, subresource=stats)"
```

- Ref issue
`kubernetes-incubator/metrics-server#77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment