New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm in self-hosted mode (1.6.1) fails to startup scheduler #8

Closed
rhuss opened this Issue Apr 4, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@rhuss
Copy link

rhuss commented Apr 4, 2017

I just did a super fresh install of everything but now hit the following issue with kubeadm 1.6.1

A

KUBE_HYPERKUBE_IMAGE=luxas/hyperkube:v1.6.0-and-PR-42911 kubeadm init --config /etc/kubernetes/kubeadm.yml

with /etc/kubernetes/kubeadm.yml (from the README except an extra predefined token)

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
token: "3d63a1.d93b1c68af864b80"
networking:
  podSubnet: "10.1.0.0/16"
#  serviceSubnet: "10.200.100.0/24"
controllerManagerExtraArgs:
  controllers: "*,-persistentvolume-binder,bootstrapsigner,tokencleaner"
  horizontal-pod-autoscaler-use-rest-clients: "true"
  horizontal-pod-autoscaler-sync-period: "10s"
  node-monitor-grace-period: "10s"
apiServerExtraArgs:
  runtime-config: "api/all=true"
  feature-gates: "TaintBasedEvictions=true"
  proxy-client-cert-file: "/etc/kubernetes/pki/front-proxy-client.crt"
  proxy-client-key-file: "/etc/kubernetes/pki/front-proxy-client.key"
selfHosted: true

leads to

root@n0:/home/pi# KUBE_HYPERKUBE_IMAGE=luxas/hyperkube:v1.6.0-and-PR-42911 kubeadm init --config /etc/kubernetes/kubeadm.yml
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.0
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [n0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.23.200]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 38.382559 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 8.119716 seconds
[self-hosted] Creating self-hosted control plane...
[self-hosted] kube-apiserver DaemonSet current=0, desired=0
[self-hosted] Found 0 self-hosted-kube-apiserver pods
[self-hosted] Found 0 self-hosted-kube-apiserver pods
[self-hosted] Found 0 self-hosted-kube-apiserver pods
....
[self-hosted] Found 1 self-hosted-kube-apiserver pods
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Pending
[self-hosted] Found 1 self-hosted-kube-apiserver pods
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Pending
[self-hosted] Found 1 self-hosted-kube-apiserver pods
.....
[self-hosted] Pod self-hosted-kube-apiserver-jx8s8 status: Running
[apiclient] All control plane components are healthy after 19.398582 seconds
[self-hosted] self-hosted kube-apiserver ready after 41.020531 seconds
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 0 self-hosted-kube-scheduler pods
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
[self-hosted] Pod self-hosted-kube-scheduler-2660681130-r0php status: Pending
[self-hosted] Found 1 self-hosted-kube-scheduler pods
.....
(never ending)

which indicates that the scheduler pod doesn't come up. And indeed the problem is that the scheduler pod cannot be scheduled:

root@n0:/home/pi# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   etcd-n0                                       1/1       Running   0          1m
kube-system   kube-controller-manager-n0                    1/1       Running   0          22s
kube-system   kube-scheduler-n0                             1/1       Running   1          1m
kube-system   self-hosted-kube-apiserver-jx8s8              1/1       Running   0          2m
kube-system   self-hosted-kube-scheduler-2660681130-r0php   0/1       Pending   0          1m

root@n0:/home/pi# kubectl describe pod self-hosted-kube-scheduler-2660681130-r0php --namespace=kube-system
Name:		self-hosted-kube-scheduler-2660681130-r0php
Namespace:	kube-system
Node:		/
Labels:		component=kube-scheduler
		k8s-app=self-hosted-kube-scheduler
		pod-template-hash=2660681130
		tier=control-plane
Annotations:	kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"self-hosted-kube-scheduler-2660681130","uid":"449d6cb5-1917-...
Status:		Pending
IP:
Controllers:	ReplicaSet/self-hosted-kube-scheduler-2660681130
Containers:
  self-hosted-kube-scheduler:
    Image:	luxas/hyperkube:v1.6.0-and-PR-42911
    Port:
    Command:
      /hyperkube
      scheduler
      --address=127.0.0.1
      --leader-elect=true
      --kubeconfig=/etc/kubernetes/scheduler.conf
    Requests:
      cpu:		100m
    Liveness:		http-get http://127.0.0.1:10251/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:	<none>
    Mounts:
      /etc/kubernetes/ from k8s (ro)
      /etc/ssl/certs from certs (rw)
      /var/lock from var-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4szsp (ro)
Conditions:
  Type		Status
  PodScheduled 	False
Volumes:
  k8s:
    Type:	HostPath (bare host directory volume)
    Path:	/etc/kubernetes
  certs:
    Type:	HostPath (bare host directory volume)
    Path:	/etc/ssl/certs
  var-lock:
    Type:	HostPath (bare host directory volume)
    Path:	/var/lock
  default-token-4szsp:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-4szsp
    Optional:	false
QoS Class:	Burstable
Node-Selectors:	node-role.kubernetes.io/master=
Tolerations:	node-role.kubernetes.io/master=:NoSchedule
		node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
		node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  1m		42s		8	default-scheduler			Warning		FailedScheduling	no nodes available to schedule pods

Looks like a catch-22 ;-)

@rhuss

This comment has been minimized.

Copy link
Author

rhuss commented Apr 4, 2017

these are the logs of the Docker container:

root@n0:/home/pi# docker logs 9552a3fccc4b
I0404 09:16:00.523415       1 leaderelection.go:179] attempting to acquire leader lease...
I0404 09:16:01.711489       1 leaderelection.go:189] successfully acquired lease kube-system/kube-scheduler
I0404 09:16:01.714291       1 event.go:217] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"2907d1f8-1917-11e7-9f3c-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"388", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' n0 became leader
I0404 09:16:01.715333       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"362", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
I0404 09:16:01.751626       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
W0404 09:16:01.751239       1 factory.go:533] Request for pod kube-system/self-hosted-kube-scheduler-2660681130-r0php already in flight, abandoning
I0404 09:16:02.734193       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
I0404 09:16:04.757063       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"self-hosted-kube-scheduler-2660681130-r0php", UID:"4671c05b-1917-11e7-8dc6-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"389", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' no nodes available to schedule pods
....
@rhuss

This comment has been minimized.

Copy link
Author

rhuss commented Apr 4, 2017

Maybe also interesting that there is also a stopped scheduler container these logs:

Full log
docker logs d59b24c9d1a1
E0404 09:14:27.925556       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.927174       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
I0404 09:14:27.928893       1 leaderelection.go:179] attempting to acquire leader lease...
E0404 09:14:27.932526       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.933367       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.935295       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.938076       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.939356       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.941254       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.942252       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:27.943986       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.930698       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.933691       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.934629       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.937206       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.939908       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.942690       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.942680       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.944690       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:28.946494       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.933107       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.936630       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.937594       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.939295       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.943207       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.944911       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.946020       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.948113       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:29.948132       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.935302       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.938779       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.940880       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.941455       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.946629       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.947426       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.949145       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.950186       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.951138       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:30.973177       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.937689       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.940800       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.942910       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.944463       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.948670       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.949826       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.951432       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.953286       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:31.953504       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.939885       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.942699       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.945117       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.946964       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.951129       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.953431       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.955548       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.956720       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:32.957001       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.942088       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.944747       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.947487       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.949991       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.952923       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.955752       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.957823       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.959875       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:33.959875       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.944226       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.946945       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.949646       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.951733       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.954721       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.957681       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.959827       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.961891       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:34.963770       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.372042       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.946302       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.948921       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.951742       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.953808       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.956664       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.959510       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.961663       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.963735       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:35.965944       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.948552       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.950985       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.953846       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.955906       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.958620       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.961310       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.963601       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.965694       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:36.968070       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.950854       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.953029       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.957571       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.957901       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.960660       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.963205       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.965626       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.968097       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:37.969899       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.402619       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.953136       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.955059       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.960051       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.961204       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.963148       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.965336       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.967653       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.970227       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:38.972056       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.957319       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.958701       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.963426       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.964318       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.965881       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.968060       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.969932       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.972155       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:39.974368       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.959180       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.961141       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.965715       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.966965       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.968548       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.970307       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.971894       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.973985       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:40.976628       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.961205       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.963975       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.968542       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.969904       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.970593       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.972589       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.973810       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.975912       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:41.978747       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.278273       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.963240       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.966679       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.971044       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.973321       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.973520       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.976741       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.978891       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.980871       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:42.981876       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.965190       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.969088       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.973076       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.975219       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.976155       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.981512       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.982122       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.983368       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:43.984129       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.969243       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.971918       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.977625       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.975556       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.979487       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.989462       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.991187       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.991187       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:44.992362       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.971443       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.974337       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.979647       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.982397       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.987661       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.991601       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.993840       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.995735       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:45.997787       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:46.004426       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:14:48.915127       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions)
E0404 09:14:48.915812       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:14:48.942015       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers)
E0404 09:14:48.942507       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:14:48.942015       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes)
E0404 09:14:48.942256       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes)
E0404 09:14:48.948150       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services)
E0404 09:14:48.949463       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps)
E0404 09:14:48.958913       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims)
E0404 09:14:49.415261       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler)
E0404 09:14:49.931453       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions)
E0404 09:14:49.932451       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:14:49.965471       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers)
E0404 09:14:49.971820       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes)
E0404 09:14:49.994315       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps)
E0404 09:14:49.999929       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:14:50.018599       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services)
E0404 09:14:50.019916       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes)
E0404 09:14:50.027406       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims)
I0404 09:14:52.155166       1 leaderelection.go:189] successfully acquired lease kube-system/kube-scheduler
I0404 09:14:52.155917       1 event.go:217] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"2907d1f8-1917-11e7-9f3c-b827ebadcfc2", APIVersion:"v1", ResourceVersion:"108", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' n0 became leader
E0404 09:15:18.992518       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=1&timeoutSeconds=331&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:18.995190       1 reflector.go:304] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to watch *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=1&timeoutSeconds=522&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.021403       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=1&timeoutSeconds=320&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.032207       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=1&timeoutSeconds=486&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.034051       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=1&timeoutSeconds=514&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.035817       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=344&timeoutSeconds=464&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.039588       1 reflector.go:304] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to watch *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=347&timeoutSeconds=428&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.041034       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=1&timeoutSeconds=370&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.058208       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.059912       1 reflector.go:304] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to watch *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=29&timeoutSeconds=340&watch=true: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.996723       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:19.999419       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.024563       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.037968       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.042794       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.035975       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.049952       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.059146       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:20.072081       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.002340       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.007076       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.017813       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.026837       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.039929       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.046970       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.047142       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.051713       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.063411       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:21.077679       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.008886       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.015563       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.033615       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.043228       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.049021       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.051081       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.055648       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.067152       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:22.082464       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.019035       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.023464       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.024604       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.035792       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.045297       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.054780       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.056666       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.058399       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.069641       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:23.086056       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.028056       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.029482       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.039431       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.048866       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.058563       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.060888       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.061258       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.072249       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:24.089485       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.024785       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get https://192.168.23.200:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.031564       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.035043       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.041539       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: Get https://192.168.23.200:6443/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.050891       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: Get https://192.168.23.200:6443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.064149       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.064748       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: Get https://192.168.23.200:6443/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.066112       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: Get https://192.168.23.200:6443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.076926       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: Get https://192.168.23.200:6443/api/v1/persistentvolumes?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:25.091848       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: Get https://192.168.23.200:6443/api/v1/services?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:26.037657       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: Get https://192.168.23.200:6443/apis/apps/v1beta1/statefulsets?resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:26.038807       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: Failed to list *v1.Pod: Get https://192.168.23.200:6443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 192.168.23.200:6443: getsockopt: connection refused
E0404 09:15:33.796928       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services)
E0404 09:15:33.797857       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes)
E0404 09:15:33.807645       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:15:33.964452       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims)
E0404 09:15:33.964748       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes)
E0404 09:15:34.647041       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.ReplicationController: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope. (get replicationcontrollers)
E0404 09:15:34.704806       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.ReplicaSet: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope. (get replicasets.extensions)
E0404 09:15:35.027778       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1beta1.StatefulSet: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope. (get statefulsets.apps)
E0404 09:15:35.028179       1 reflector.go:201] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454: Failed to list *v1.Pod: User "system:kube-scheduler" cannot list pods at the cluster scope. (get pods)
E0404 09:15:35.028430       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler)
E0404 09:15:35.029417       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Service: User "system:kube-scheduler" cannot list services at the cluster scope. (get services)
E0404 09:15:35.029694       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolume: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope. (get persistentvolumes)
E0404 09:15:35.263828       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.PersistentVolumeClaim: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope. (get persistentvolumeclaims)
E0404 09:15:35.264118       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: User "system:kube-scheduler" cannot get endpoints in the namespace "kube-system". (get endpoints kube-scheduler)
E0404 09:15:35.265475       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: Failed to list *v1.Node: User "system:kube-scheduler" cannot list nodes at the cluster scope. (get nodes)
E0404 09:15:35.265388       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'n0 stopped leading'
I0404 09:15:35.267394       1 leaderelection.go:208] failed to renew lease kube-system/kube-scheduler
F0404 09:15:35.268121       1 server.go:134] lost master
@rhuss

This comment has been minimized.

Copy link
Author

rhuss commented Apr 5, 2017

Actually this is only an issue when running in self-hosted mode. Without selfHosted: true in the config file it works.

Current status: I'm now back using plain kubeadm 1.6.1 (without your custom hyperkube image) and disabled some features in the config file (like proxy-client-cert-file) since these are not available in for vanilla apiserver 1.6.0.

Now it works, and even weave is happy (because of different machine-ids now).

Now going to try PVs and Ingress (as I guess they don't require your customisations in the adapter hyperkube image ...)

Let me know if I can help further in evaluating the issue with the self-hosted mode.

@rhuss rhuss changed the title kubeadm init (1.6.1) fails to startup scheduler kubeadm in self-hosted mode (1.6.1) fails to startup scheduler Apr 5, 2017

@luxas

This comment has been minimized.

Copy link
Owner

luxas commented Apr 5, 2017

Known issue. Since the kubelet behaviour changed drastically between beta.4 and rc.1-ish and kubeadm broke fatally as a consequence, we fixed that by breaking self-hosting only in v1.6.1 ;)

It's on my list to do pretty soon, but well, school ;)

As you noted, it will just work fine to disable proxy* and self-hosting if you don't want the API aggregation thing.

Awesome to see you're working on this. When you find flaws, feel free to do a PR as well 👍

@rhuss

This comment has been minimized.

Copy link
Author

rhuss commented Apr 5, 2017

Thanks for the info and for your awesome work on kubeadm and kubernetes in general.

np at all, take your time. School is important. still ;-)

'hope to contribute more, still looking around to get a feeling of the project(s). And yes, spare time is an issue here, too :)

@rhuss rhuss closed this Apr 5, 2017

@luxas

This comment has been minimized.

Copy link
Owner

luxas commented Apr 30, 2017

I could now move from self-hosting to using static pods as normal, so this should be ok now 👍
Thanks ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment