no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service

[TreHack]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
  NAME="Debian GNU/Linux"
  VERSION_ID="9"
  VERSION="9 (stretch)"
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"
• Kernel (e.g. uname -a):
  Linux top-unknown139863.i.nease.net 4.9.0-4-amd64 kubeadm join on slave node fails preflight checks #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux
• Others:

What happened?

root@top-unknown139863:~# kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=0.0.0.0
[init] using Kubernetes version: v1.11.2
[preflight] running pre-flight checks
I0810 13:04:57.272222   21281 kernel_validator.go:81] Validating kernel version
I0810 13:04:57.272369   21281 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [top-unknown139863.i.nease.net kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.202.15.11]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [top-unknown139863.i.nease.net localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [top-unknown139863.i.nease.net localhost] and IPs [10.202.15.11 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled

                Unfortunately, an error has occurred:
                        timed out waiting for the condition

                This error is likely caused by:
                        - The kubelet is not running
                        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
                        - No internet connection is available so the kubelet cannot pull or find the following control plane images:
                                - k8s.gcr.io/kube-apiserver-amd64:v1.11.2
                                - k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
                                - k8s.gcr.io/kube-scheduler-amd64:v1.11.2
                                - k8s.gcr.io/etcd-amd64:3.2.18
                                - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
                                  are downloaded locally and cached.

                If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                        - 'systemctl status kubelet'
                        - 'journalctl -xeu kubelet'

                Additionally, a control plane component may have crashed or exited when started by the container runtime.
                To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
                Here is one example how you may list all Kubernetes containers running in docker:
                        - 'docker ps -a | grep kube | grep -v pause'
                        Once you have found the failing container, you can inspect its logs with:
                        - 'docker logs CONTAINERID'

root@top-unknown139863:~# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2018-08-10 13:05:30 HKT; 20min ago
     Docs: http://kubernetes.io/docs/
 Main PID: 21766 (kubelet)
    Tasks: 19 (limit: 4915)
   Memory: 41.6M
      CPU: 26.260s
   CGroup: /system.slice/kubelet.service
           └─21766 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin

Aug 10 13:25:42 top-unknown139863.i.nease.net kubelet[21766]: I0810 13:25:42.298624   21766 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 10 13:25:46 top-unknown139863.i.nease.net kubelet[21766]: W0810 13:25:46.031381   21766 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Aug 10 13:25:46 top-unknown139863.i.nease.net kubelet[21766]: E0810 13:25:46.031644   21766 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne
Aug 10 13:25:50 top-unknown139863.i.nease.net kubelet[21766]: E0810 13:25:50.617126   21766 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "top-unknown13
Aug 10 13:25:51 top-unknown139863.i.nease.net kubelet[21766]: W0810 13:25:51.032972   21766 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Aug 10 13:25:51 top-unknown139863.i.nease.net kubelet[21766]: E0810 13:25:51.033235   21766 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne
Aug 10 13:25:51 top-unknown139863.i.nease.net kubelet[21766]: I0810 13:25:51.298705   21766 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 10 13:25:53 top-unknown139863.i.nease.net kubelet[21766]: E0810 13:25:53.288635   21766 kubelet_node_status.go:103] Unable to register node "top-unknown139863.i.nease.net" with API server: Post https://10.202
Aug 10 13:25:56 top-unknown139863.i.nease.net kubelet[21766]: W0810 13:25:56.034830   21766 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Aug 10 13:25:56 top-unknown139863.i.nease.net kubelet[21766]: E0810 13:25:56.035200   21766 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne
lines 1-23/23 (END)

root@top-unknown139863:~# docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-controller-manager-amd64   v1.11.2             38521457c799        2 days ago          155 MB
k8s.gcr.io/kube-apiserver-amd64            v1.11.2             821507941e9c        2 days ago          187 MB
k8s.gcr.io/kube-proxy-amd64                v1.11.2             46a3cd725628        2 days ago          97.8 MB
k8s.gcr.io/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        2 days ago          56.8 MB
k8s.gcr.io/coredns                         1.1.3               b3b94275d97c        2 months ago        45.6 MB
k8s.gcr.io/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219 MB
k8s.gcr.io/pause                           3.1                 da86e6ba6ca1        7 months ago        742 kB

root@top-unknown139863:~# docker ps -a | grep kube | grep -v pause
51b23382087d        821507941e9c           "kube-apiserver --..."   About a minute ago   Up About a minute                              k8s_kube-apiserver_kube-apiserver-top-unknown139863.i.nease.net_kube-system_09c8a4c6557753a1631dad18a0a0fdbc_7
1cc128e59bdd        821507941e9c           "kube-apiserver --..."   7 minutes ago        Exited (0) 4 minutes ago                       k8s_kube-apiserver_kube-apiserver-top-unknown139863.i.nease.net_kube-system_09c8a4c6557753a1631dad18a0a0fdbc_6
40ff34a83c2e        37a1403e6c1a           "kube-scheduler --..."   21 minutes ago       Up 21 minutes                                  k8s_kube-scheduler_kube-scheduler-top-unknown139863.i.nease.net_kube-system_d3639452e9a6afd70dcc980e757a0ea0_0
35a2483248c6        38521457c799           "kube-controller-m..."   21 minutes ago       Up 21 minutes                                  k8s_kube-controller-manager_kube-controller-manager-top-unknown139863.i.nease.net_kube-system_6b6f17a9cf2ffebcb71c9a65992a8c02_0
76223f733a7a        b8df3b177be2           "etcd --advertise-..."   21 minutes ago       Up 21 minutes                                  k8s_etcd_etcd-top-unknown139863.i.nease.net_kube-system_281634f4a0fdec085e04c90414813849_0

root@top-unknown139863:~# docker logs 51b23382087d
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0810 05:24:57.155678       1 server.go:703] external host was not specified, using 10.202.15.11
I0810 05:24:57.155961       1 server.go:145] Version: v1.11.2
I0810 05:24:57.892799       1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0810 05:24:57.892834       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0810 05:24:57.893617       1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0810 05:24:57.893629       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0810 05:24:57.968109       1 master.go:234] Using reconciler: lease
W0810 05:25:00.533541       1 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0810 05:25:01.144718       1 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0810 05:25:01.155884       1 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0810 05:25:01.210722       1 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0810 05:25:02.273121       1 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/10 05:25:02 log.go:33: [restful/swagger] listing is available at https://10.202.15.11:6443/swaggerapi
[restful] 2018/08/10 05:25:02 log.go:33: [restful/swagger] https://10.202.15.11:6443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/10 05:25:04 log.go:33: [restful/swagger] listing is available at https://10.202.15.11:6443/swaggerapi
[restful] 2018/08/10 05:25:04 log.go:33: [restful/swagger] https://10.202.15.11:6443/swaggerui/ is mapped to folder /swagger-ui/
I0810 05:25:04.092968       1 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0810 05:25:04.093007       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0810 05:25:07.983549       1 serve.go:96] Serving securely on 0.0.0.0:6443
I0810 05:25:07.983732       1 crd_finalizer.go:242] Starting CRDFinalizer
I0810 05:25:07.984553       1 available_controller.go:278] Starting AvailableConditionController
I0810 05:25:07.984620       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0810 05:25:07.984670       1 controller.go:84] Starting OpenAPI AggregationController
I0810 05:25:07.985332       1 autoregister_controller.go:136] Starting autoregister controller
I0810 05:25:07.985355       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0810 05:25:07.985522       1 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0810 05:25:07.985540       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0810 05:25:07.986374       1 customresource_discovery_controller.go:199] Starting DiscoveryController
I0810 05:25:07.986474       1 naming_controller.go:284] Starting NamingConditionController
I0810 05:25:07.986595       1 establishing_controller.go:73] Starting EstablishingController
I0810 05:25:07.989216       1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0810 05:25:07.989309       1 controller_utils.go:1025] Waiting for caches to sync for crd-autoregister controller
I0810 05:25:08.084844       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0810 05:25:08.085552       1 cache.go:39] Caches are synced for autoregister controller
I0810 05:25:08.088410       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0810 05:25:08.090052       1 controller_utils.go:1032] Caches are synced for crd-autoregister controller
I0810 05:25:08.992575       1 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
I0810 05:27:21.631840       1 controller.go:158] Shutting down kubernetes service endpoint reconciler
I0810 05:27:21.632166       1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I0810 05:27:21.632221       1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0810 05:27:21.632252       1 naming_controller.go:295] Shutting down NamingConditionController
I0810 05:27:21.632271       1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I0810 05:27:21.632296       1 establishing_controller.go:84] Shutting down EstablishingController
I0810 05:27:21.632321       1 autoregister_controller.go:160] Shutting down autoregister controller
I0810 05:27:21.632344       1 available_controller.go:290] Shutting down AvailableConditionController
I0810 05:27:21.632363       1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0810 05:27:21.632635       1 controller.go:90] Shutting down OpenAPI AggregationController
I0810 05:27:21.632735       1 serve.go:136] Stopped listening on 0.0.0.0:6443
E0810 05:27:21.696607       1 controller.go:160] no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service

root@top-unknown139863:~# ps -ef | grep kube
root     21766     1  1 13:05 ?        00:00:28 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
root     22044 22025  1 13:05 ?        00:00:22 etcd --advertise-client-urls=https://127.0.0.1:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://127.0.0.1:2380 --initial-cluster=top-unknown139863.i.nease.net=https://127.0.0.1:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379 --listen-peer-urls=https://127.0.0.1:2380 --name=top-unknown139863.i.nease.net --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root     22144 22125  0 13:05 ?        00:00:00 kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=192.168.0.0/16 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --node-cidr-mask-size=24 --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root     22196 22167  0 13:05 ?        00:00:02 kube-scheduler --address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root     29374 29355 14 13:27 ?        00:00:15 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=10.202.15.11 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=PersistentVolumeLabel --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root     29917 20575  0 13:29 pts/0    00:00:00 grep kube

What you expected to happen?

kubeadm init success

It seems that the api-server can't start, what‘s the mean of no master IPs were listed in storage?

[neolit123]

if you change --apiserver-advertise-address=0.0.0.0 to the public address does it work?
this could be due to a proxy issue, in case you are using a proxy, that is.

[TreHack]

@neolit123 It doesn't work to change --apiserver-advertise-address=0.0.0.0 to the public address , the same log to the above.

[abha10]

silly check but have you set setenforce=0 ??

[lijiaocn]

I have the same problem.

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

who have the soluation....

[lijiaocn]

I just add a config line in "/etc/kubernetes/manifests/kube-apiserver.yaml":

• --endpoint-reconciler-type=none

the problem disappeared.

default value for --endpoint-reconciler-type is lease. there may be a bug in apiserver when use the default config.

[timothysc]

I'm going to close this issue and if folks want to please open an issue in the main repo.

[WilliamThh]

I have the same problem.
when I upgrading kubeadm clusters from v1.10 to v1.11 .
According to the course：
https://v1-12.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/
when I run systemctl restart kubelet，apiserver will be shutting down and I find same problem.
I try to change parameter --endpoint-reconciler-type=none but It doesn't work.
I don't know why ...

[fuzhibo]

I just add a config line in "/etc/kubernetes/manifests/kube-apiserver.yaml":

• --endpoint-reconciler-type=none

the problem disappeared.

default value for --endpoint-reconciler-type is lease. there may be a bug in apiserver when use the default config.

We should not use --endpoint-reconciler-type=none now to solve if api server has this error, --endpoint-reconciler-type is very important for auto create Endpoint for Service, keep it as default. When we face that error, it may seems etcd server is unavailable, check if etcd not work, maybe containerd's config.toml with SystemdCgroup = false will triger the issue.