k3s kubectl top pods errors

[denwork]

INSTALL:
HA externaldb : etcd
metrics-server: rancher/metrics-server:v0.3.6

shell
curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh |K3S_DATASTORE_ENDPOINT='https://master1.example.ex:2379,https://master3.example.ex:2379,https://master2.example.ex:2379' INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC="server --docker --flannel-backend='none' --datastore-cafile=/etc/kubernetes/cert/ca.pem --datastore-certfile=/etc/etcd/cert/etcd.pem --datastore-keyfile=/etc/etcd/cert/etcd-key.pem" sh -
STATUS:

[root@master1 ~]# kubectl top nodes
NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master1.example.ex   520m         6%     3756Mi          23%       
master2.example.ex   597m         7%     5952Mi          37%       
master3.example.ex   493m         6%     3496Mi          22%       
slave1.example.ex    392m         4%     2593Mi          16%       
slave2.example.ex    298m         3%     2827Mi          17%       
[root@master1 ~]# kubectl top pods
W0318 17:16:57.565391   14590 top_pod.go:266] Metrics not available for pod default/nginx-54d6fd96b9-chl25, age: 25h49m33.565360117s
error: Metrics not available for pod default/nginx-54d6fd96b9-chl25, age: 25h49m33.565360117s

container logs:
[root@master1 ~]# kubectl logs -n kube-system metrics-server-6d684c7b5-5gdm4
I0312 12:19:35.789576 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0312 12:19:36.811159 1 secure_serving.go:116] Serving securely on [::]:443
I0312 13:17:03.151023 1 log.go:172] http: TLS handshake error from 10.50.188.192:46578: remote error: tls: unknown certificate authority
I0312 13:17:17.151274 1 log.go:172] http: TLS handshake error from 172.21.1.83:15178: remote error: tls: unknown certificate authority
I0312 13:17:22.367831 1 log.go:172] http: TLS handshake error from 10.50.166.128:58558: remote error: tls: unknown certificate authority
E0314 05:14:18.244733 1 reflector.go:270] k8s.io/client-go/informers/factory.go:133: Failed to watch *v1.Node: Get https://10.43.0.1:443/api/v1/nodes?resourceVersion=804642&timeout=9m6s&timeoutSeconds=546&watch=true: dial tcp 10.43.0.1:443: connect: connection refused
E0314 05:14:18.593782 1 reflector.go:270] k8s.io/client-go/informers/factory.go:133: Failed to watch *v1.Pod: Get https://10.43.0.1:443/api/v1/pods?resourceVersion=733192&timeout=9m44s&timeoutSeconds=584&watch=true: dial tcp 10.43.0.1:443: connect: connection refused
E0314 05:14:19.246268 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get https://10.43.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused
E0314 05:14:20.247571 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get https://10.43.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused
E0314 05:14:21.248977 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get https://10.43.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused
E0314 05:14:22.250284 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get https://10.43.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused
E0315 12:50:25.949048 1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6bc84865b8-f8kjx: no metrics known for pod
E0315 12:50:40.959085 1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6bc84865b8-f8kjx: no metrics known for pod
E0315 12:50:56.860417 1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6bc84865b8-f8kjx: no metrics known for pod

Who can help me ?

[brandond]

Have you tried using the built-in containerd instead of docker? Is there any particular reason you're using docker instead?

[denwork]

@brandond
The cluster cannot be started with containerd and the images produced are all built with docker.
I have tried using metrics-server-deployment.yaml in k8s cluster instead of k3s metrics-server-deployment.But the problem has not changed.

[denwork]

@brandond I've already tried containerd instead of docker. kubectl top pod is that OK.
The results are the same after multiple tests in multiple environments, does this mean that using docker is problematic?

[hongkunyoo]

I'm having a same issue, same error logs.
metrics-server works on containerd, while fails on docker.
Is there a plan to fix it on docker? It would be great if so.

[loganhz]

Steps to reproduce:

1. Launch Ubuntu 18.04 on AWS
2. sudo apt update
3. sudo apt install docker.io (PS: Docker CE 19.03.6 installed)
4. curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --docker" sh -

Results:

kubectl top nodes works
kubectl top pods --all-namespaces doesn't work.

W0408 14:33:01.559059    9179 top_pod.go:266] Metrics not available for pod default/nginx-86c57db685-7f8pv, age: 8m17.559045604s
error: Metrics not available for pod default/nginx-86c57db685-7f8pv, age: 8m17.559045604s

More Info:

Cannot reproduce it in Docker CE 17.03, so it's related to docker version.

Expected:

K3s should compatible with latest Docker version 19.03.6 .

[ShylajaDevadiga]

Verified metrics is reported using the fix #1627
Docker version 17.03, Ubuntu 16.04.6 LTS

 k3s -v
k3s version v1.17.4+k3s-48eb3d81 (48eb3d81)
sudo  k3s kubectl top pods -A
NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)   
kube-system   coredns-7b8c5b77b4-dzk7q                  2m           7Mi             
kube-system   local-path-provisioner-58dbf4c86c-rdscs   2m           6Mi             
kube-system   metrics-server-59b5bcbc44-swh7f           1m           9Mi             
kube-system   svclb-traefik-9phgw                       0m           1Mi             
kube-system   traefik-5f7b78ff-gkhg4                    3m           11Mi 

sudo docker version|grep Version
 Version:      17.03.3-ce

And on Docker version 18.03, 19.06, 19.03, Ubuntu 18.04.3 LTS

k3s -v
k3s version v1.17.4+k3s-48eb3d81 (48eb3d81)

sudo kubectl top  pods -A
NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)   
kube-system   coredns-7b8c5b77b4-jpq26                  2m           7Mi             
kube-system   local-path-provisioner-58dbf4c86c-7z8pk   2m           6Mi             
kube-system   metrics-server-59b5bcbc44-4vtmz           1m           9Mi             
kube-system   svclb-traefik-zqklr                       0m           1Mi             
kube-system   traefik-5f7b78ff-2hh4m                    3m           12Mi

sudo docker version  |grep Version
 Version:           19.03.8