Failed to list *v1.Node: Unauthorized

[wrightxu]

在vmware创建了5个虚拟机测试，前3个节点挂上了，后2个没有部署上，报错：
5月 07 17:44:15 node4 kubelet[52929]: I0507 17:44:15.947332 52929 kubelet_node_status.go:452] Recording NodeHasNoDiskPressure event message for node 192.168.100.114
5月 07 17:44:15 node4 kubelet[52929]: I0507 17:44:15.947337 52929 kubelet_node_status.go:452] Recording NodeHasSufficientPID event message for node 192.168.100.114
5月 07 17:44:15 node4 kubelet[52929]: I0507 17:44:15.947348 52929 kubelet_node_status.go:93] Attempting to register node 192.168.100.114
5月 07 17:44:15 node4 kubelet[52929]: E0507 17:44:15.950074 52929 kubelet_node_status.go:117] Unable to register node "192.168.100.114" with API server: Unauthorized
5月 07 17:44:16 node4 kubelet[52929]: E0507 17:44:16.377891 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Unauthorized
5月 07 17:44:16 node4 kubelet[52929]: E0507 17:44:16.425108 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized
5月 07 17:44:16 node4 kubelet[52929]: E0507 17:44:16.445579 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
5月 07 17:44:17 node4 kubelet[52929]: E0507 17:44:17.380186 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Unauthorized
5月 07 17:44:17 node4 kubelet[52929]: E0507 17:44:17.427314 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized
5月 07 17:44:17 node4 kubelet[52929]: E0507 17:44:17.447967 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
5月 07 17:44:18 node4 kubelet[52929]: E0507 17:44:18.381821 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Failed to list *v1.Node: Unauthorized
5月 07 17:44:18 node4 kubelet[52929]: E0507 17:44:18.428575 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized
5月 07 17:44:18 node4 kubelet[52929]: E0507 17:44:18.451040 52929 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized

[wrightxu]

hosts:

集群部署节点：一般为运行ansible 脚本的节点

变量 NTP_ENABLED (=yes/no) 设置集群是否安装 chrony 时间同步

[deploy]
192.168.100.111 NTP_ENABLED=no

etcd集群请提供如下NODE_NAME，注意etcd集群必须是1,3,5,7...奇数个节点

[etcd]
192.168.100.111 NODE_NAME=node1
192.168.100.112 NODE_NAME=node2
192.168.100.113 NODE_NAME=node3

[kube-master]
192.168.100.111

[kube-node]
192.168.100.112
192.168.100.113
192.168.100.114
192.168.100.115

参数 NEW_INSTALL：yes表示新建，no表示使用已有harbor服务器

如果不使用域名，可以设置 HARBOR_DOMAIN=""

[harbor]
#192.168.1.8 HARBOR_DOMAIN="harbor.yourdomain.com" NEW_INSTALL=no

【可选】外部负载均衡，用于自有环境负载转发 NodePort 暴露的服务等

[ex-lb]
#192.168.1.6 LB_ROLE=backup EX_VIP=192.168.1.250
#192.168.1.7 LB_ROLE=master EX_VIP=192.168.1.250

[all:vars]

---------集群主要参数---------------

#集群部署模式：allinone, single-master, multi-master
DEPLOY_MODE=single-master

[wrightxu]

[root@host1 ansible]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.100.111 Ready,SchedulingDisabled master 4h v1.10.13
192.168.100.112 Ready node 4h v1.10.13
192.168.100.113 Ready node 4h v1.10.13

[gjmzj]

检查下失败节点的 /etc/kubernetes 目录看看 kubelet相关的证书，配置在不在；

[root@kube142 ~]# ll /etc/kubernetes/
total 16
-rw------- 1 root root 6354 May  8 15:30 kubelet.kubeconfig
-rw------- 1 root root 6244 May  4 22:57 kube-proxy.kubeconfig
drwxr-xr-x 2 root root  145 May  8 15:30 ssl
[root@kube142 ~]# ll /etc/kubernetes/ssl/
total 28
-rw-r----- 1 root root  294 May  4 22:53 ca-config.json
-rw------- 1 root root 1679 May  4 22:53 ca-key.pem
-rw-r----- 1 root root 1350 May  4 22:53 ca.pem
-rw-r--r-- 1 root root 1082 May  8 15:30 kubelet.csr
-rw-r--r-- 1 root root  283 May  8 15:30 kubelet-csr.json
-rw------- 1 root root 1679 May  8 15:30 kubelet-key.pem
-rw-r--r-- 1 root root 1452 May  8 15:30 kubelet.pem

[wrightxu]

群主远程登陆处理，发现是每个主机的时间不一致导致认证失败。
我将群主的方法简单整理如下：
一、定位问题
1、master上查apiserver日志：
systemctl status kube-apiserver -l
2、kubectl get cs
发现证书失效或者无效
3、查主机时间
ansible all -m shell -a 'date'
发现时间差一天左右
二、处理：
调整host，修改为ntp
[deploy]
192.168.100.111 NTP_ENABLED=yes