BUG REPORT: Clean new install on CentOS7x kubelet fails to start with error: kubelet: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory

[ernestgwilsonii]

BUG REPORT

Clean / new install(s) on CentOS7x kubelet fails to start with error:
kubelet: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory

Kubernetes version (use kubectl version):
kubelet --version
Kubernetes v1.4.1

kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Environment:

• Cloud provider or hardware configuration:
  New clean install of Centos7x

• OS (e.g. from /etc/os-release):
  cat /etc/redhat-release
  CentOS Linux release 7.3.1611 (Core)

• Kernel (e.g. uname -a):
  uname -a
  Linux VA1EWTKTC02 3.10.0-327.18.2.el7.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

What happened:
Following official guide here:
https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/
Near the top section of the guide, when using CentOS7x and the command outlined in the guide is:
systemctl enable kubelet && systemctl start kubelet
kublet fails to start and an error is logged:
kubelet: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory

What you expected to happen:
systemctl start kubelet
should have started kubelet

How to reproduce it (as minimally and precisely as possible):
Start with any typical minimal clean CentOS7x OS install.
Follow this official guide:
https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/
Barely get down the page following the guide and boom, kubelet won't start.
systemctl enable kubelet && systemctl start kubelet

systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Sat 2017-02-04 15:19:24 UTC; 4s ago
Docs: https://github.com/kubernetes/kubernetes
Process: 6410 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 6410 (code=exited, status=1/FAILURE)

Anything else we need to know:
cat /var/log/messages
Feb 4 22:49:02 VA1EWTKTC02 systemd: kubelet.service holdoff time over, scheduling restart.
Feb 4 22:49:02 VA1EWTKTC02 systemd: Cannot add dependency job for unit microcode.service, ignoring: Unit is not loaded properly: Invalid argument.
Feb 4 22:49:02 VA1EWTKTC02 systemd: Started Kubernetes Kubelet Server.
Feb 4 22:49:02 VA1EWTKTC02 systemd: Starting Kubernetes Kubelet Server...
Feb 4 22:49:02 VA1EWTKTC02 kubelet: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
Feb 4 22:49:02 VA1EWTKTC02 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Feb 4 22:49:02 VA1EWTKTC02 systemd: Unit kubelet.service entered failed state.
Feb 4 22:49:02 VA1EWTKTC02 systemd: kubelet.service failed.

[aoxn]

This is predictable behavior which allow k8s installer (like kubeadm) to bootstrap kubelet.

[ernestgwilsonii]

Ok thanks, I simply ignored the error and continued on with the next step in the guide and it seemed to work.
Guide:
https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/
Next step (modified to skip the pre-flight check which also fails on CentOS):
kubeadm init --skip-preflight-checks

[alexandroscgb]

hi. I'm stuck in the part where they create the statefulset, with this following error:

Temporarily unable to list nodes (will retry)

This shows up after kubeadm init on the master. Any ideas?

[ghost]

why this is closed without any solution? what's the purpose for reporting?

[kweisamx]

I meet the same problem too, on centos 7 , the kubelet always

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2017-08-24 11:43:06 CST; 677ms ago
     Docs: http://kubernetes.io/docs/
  Process: 30471 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 30471 (code=exited, status=1/FAILURE)

[awesomemayank007]

please switchoff your swap memory i faced same problem and when i remove swap entry from my /etc/fstab file .then it worked in my case.also do swapoff

[arunm8489]

i swithedoff the swap memory.But i still faces the same problem

[arunm8489]

#yum install -y kubelet kubeadm kubectl docker
Jst try installing docker 1.12...iam not getting solution with docker 1.17ce.( i think its not supported with kubelet)
Make swap off by #swapoff -a
Now reset kubeadm by #kubeadm reset
Now try #kudeadm init
after that check #systemctl status kubelet
it will be working

[RakeshNagarajan]

even in docker Version: 1.13.1 same issue

[FlorianHeigl]

@aoxn if it's predictable, why does it log an error message? :)

[LucasSales]

Kubeadm reset

[iammrchetan]

Hi Guys,
I'm new to docker/k8s; facing the same error; kubelet is not running by "systemctl status kubelet". Could you please share how did you fix this issue?

Installed k8s following this --> https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/

[root@kekecentest04 system]# rpm -aq | grep -i kube
kubelet-1.12.2-0.x86_64
kubeadm-1.12.2-0.x86_64
kubernetes-cni-0.6.0-0.x86_64
kubectl-1.12.2-0.x86_64

CentOS Linux release 7.4.1708 (Core)
==== kubectl version ====
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

==== kubeadm version ====
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:51:33Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

==== docker version ==== installed using tarball
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:20:43 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:28:38 2018
OS/Arch: linux/amd64
Experimental: false

==== output of journalctl _SYSTEMD_UNIT=kubelet.service ====
Nov 14 10:25:53 kekecentest05 kubelet[13018]: F1114 10:25:53.426389 13018 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed

==== output while tried creating config.yaml from solution someone suggested ====
kubeadm upgrade node config --kubelet-version v1.12.2
couldn't create a Kubernetes client from file "/etc/kubernetes/kubelet.conf": failed to load admin kubeconfig [open /etc/kubernetes/kubelet.conf: no such file or directory]