kubeadm init blocks at waiting for the control plane to become ready on ubuntu 16.04

[Zjianglin]

#

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Related to or a similar discussion happened @ #33544, #33729
, #35684

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
This is a BUG REPORT

Kubernetes version (use kubectl version):

root@kubeMaster:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  No cloud provider; 2 GB Memory.100G disk.
• OS (e.g. from /etc/os-release): VERSION="16.04 LTS (Xenial Xerus)"
• Kernel (e.g. uname -a):Linux kubeMaster 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others:

root@kubeMaster:~# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.421+a6bea3d79b8bba", GitCommit:"a6bea3d79b8bbaa5e8b57482c9fff9265d402708", GitTreeState:"clean", BuildDate:"2016-11-03T06:54:50Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

What happened:
kubeadm init on mater blocks at <master/apiclient> created API client, waiting for the control plane to become ready.

What you expected to happen:
kubeadm init should finish in several minutes at most, usually less than one minute.

How to reproduce it (as minimally and precisely as possible):
Follow along side the Installing Kubernetes on Linux with kubeadm:

1. install docker: curl -sSL https://get.docker.com | sh
2. Installing kubelet and kubeadm:

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
# apt-get update
# # Install docker if you don't have it already.
# apt-get install -y docker.io
# apt-get install -y kubelet kubeadm kubectl kubernetes-cni

3. Due to GFW, I can not pull images from gcr.io. so I pull images from docker hub and tag them by myself.

images=(kube-proxy-amd64:v1.4.4 kube-discovery-amd64:1.0 kubedns-amd64:1.7 kube-scheduler-amd64:v1.4.4 kube-controller-manager-amd64:v1.4.4 kube-apiserver-amd64:v1.4.4 etcd-amd64:2.2.5 kube-dnsmasq-amd64:1.3 exechealthz-amd64:1.1 pause-amd64:3.0 kubernetes-dashboard-amd64:v1.4.1)
for imageName in ${images[@]} ; do
  docker pull mritd/$imageName
  docker tag mritd/$imageName gcr.io/google_containers/$imageName
  docker rmi mritd/$imageName
done

4. kubeadm init: kubeadm init --use-kubernetes-version v1.4.4

Anything else do we need to know:
While the kubeadm init --use-kubernetes-version v1.4.4 hangs on,

root@kubeMaster:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   etcd-kubemaster                      1/1       Running   0          21m
kube-system   kube-apiserver-kubemaster            1/1       Running   0          21m
kube-system   kube-controller-manager-kubemaster   1/1       Running   0          21m
kube-system   kube-scheduler-kubemaster            1/1       Running   0          21m

the components in control plane seems to started already.
Plus:

root@kubeMaster:~# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"} 

root@kubeMaster:~# docker images | grep gcr.io
gcr.io/google_containers/pause-amd64                     3.0                 39a06579e733        3 weeks ago         746.9 kB
gcr.io/google_containers/kubedns-amd64                   1.7                 96fdd723a50d        3 weeks ago         55.06 MB
gcr.io/google_containers/kube-dnsmasq-amd64              1.3                 2a7c0456186f        3 weeks ago         5.13 MB
gcr.io/google_containers/kube-discovery-amd64            1.0                 b873a3c6d819        3 weeks ago         134.2 MB
gcr.io/google_containers/exechealthz-amd64               1.1                 0e2effc928a5        3 weeks ago         8.332 MB
gcr.io/google_containers/etcd-amd64                      2.2.5               fbabfa0f7ac7        3 weeks ago         30.45 MB
gcr.io/google_containers/kube-scheduler-amd64            v1.4.4              ebdf41825f08        5 weeks ago         81.67 MB
gcr.io/google_containers/kube-proxy-amd64                v1.4.4              c0fcee70820a        5 weeks ago         202.7 MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.4.4              bbb0118acb7c        5 weeks ago         142 MB
gcr.io/google_containers/kube-apiserver-amd64            v1.4.4              84998e7ce18f        5 weeks ago         152.1 MB

Docker version:

root@kubeMaster:~# docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

#####Firstly, the kubeadm init blocks due to failed to pull images from gcr.io, so I pull and tag the required images before run kubeadm init. I had successfully setup a cluster in this way several weeks ago. But after some time, I reinstall(on a fresh OS) in the same way but encounter the hangs on problem again. I have tried several times. ##### Any one could help me?

[zqWu]

same problem

[himulawang]

Same problem

[Zjianglin]

I guess it was due to execute commands behind a http_proxy, try to unset http_proxy , then kubeadm reset; systemctl restart kubelet; kubeadm init?

[iming0319]

thank you @Zjianglin , you are right.
I meet the same issue and I solved this by unset the http_proxy.

[kachkaev]

Hi guys, could you please explain what do you mean to unset http_proxy? Is it an environment variable, a config parameter or something else? Cheers!

[luxas]

Please reopen this issue in the kubeadm repo in case you can reproduce this error: https://github.com/kubernetes/kubeadm/issues

Please try to use kubeadm v1.6

@kachkaev It's an environment variable. Maybe try unset HTTP_PROXY...