Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod logs don't work on some nodes #22629

Closed
pietervogelaar opened this issue Sep 4, 2019 · 5 comments

Comments

@pietervogelaar
Copy link

commented Sep 4, 2019

What kind of request is this (question/bug/enhancement/feature request):
Bug

Steps to reproduce (least amount of steps as possible):
On my laptop I have:

  • 1 Rancher VirtualBox VM
  • 3 Kubernetes sandbox cluster VirtualBox VMs

Rancher is setup with docker-machine (RancherOS) and RKE.
The sandbox cluster is setup with docker-machine (RancherOS) and the command is executed to run the agent and join the cluster.

Result:
Rancher installs the sandbox cluster on the three nodes and they all become active. If I deploy a Helm chart, pods are running fine. However on one node showing the logs of the pod just works, and on another I get:

the server could not find the requested resource ( pods/log my-app-59cdbc4754-7drz7)

If I just do docker logs <container-id> I can see the log of that pod container just fine!

If viewing the log of a pod doesn't work, then also "Exec shell" in the Rancher UI doesn't work "Closed Code: 1006".

Other details that may be helpful:
Could this be related to kubernetes/kubernetes#57359?

Environment information

  • Rancher version: v2.2.8
  • Installation option (single install/HA): HA

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported):
    Custom

  • Machine type (cloud/VM/metal) and specifications (CPU/memory):
    VM
    cpu: 1
    memory: 2048

  • Kubernetes version (use kubectl version):
    v1.15.2

  • Docker version (use docker version):
    18.09.8

@superseb

This comment has been minimized.

Copy link
Member

commented Sep 4, 2019

Please provide more information on how to reproduce and component logging involved. What hostnames are being used, what is the output of kubectl get nodes -o json, what roles are on each node, what is being logged on kube-apiserver(s) and what is logged on the kubelet of the node where the pod is running? Did you test it using the embedded kubectl or remote or both? What is output of kubectl logs when you add -v9 to it?

@pietervogelaar

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

kubectl logs example-ruby-api-95f9c4d67-hx6ls

Error from server (NotFound): the server could not find the requested resource ( pods/log example-ruby-api-95f9c4d67-hx6ls)

kubectl logs -v9 example-ruby-api-95f9c4d67-hx6ls

I0905 09:35:45.154382    1515 loader.go:359] Config loaded from file:  /host-local/cache/kubernetes/kubeconfig
I0905 09:35:45.224455    1515 round_trippers.go:419] curl -k -v -XGET  -H "Authorization: Bearer kubeconfig-user-66lw9.c-sl8lh:cwprntbbt54g7rdqghgl98qknkr69p54qzv5wbp2c5zhvp2jdjgbnr" -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.2 (linux/amd64) kubernetes/f627830" 'https://rancher.localdev/k8s/clusters/c-sl8lh/api/v1/namespaces/foo-development-early-adopters/pods/example-ruby-api-95f9c4d67-hx6ls'
I0905 09:35:45.240197    1515 round_trippers.go:438] GET https://rancher.localdev/k8s/clusters/c-sl8lh/api/v1/namespaces/foo-development-early-adopters/pods/example-ruby-api-95f9c4d67-hx6ls 200 OK in 15 milliseconds
I0905 09:35:45.240247    1515 round_trippers.go:444] Response Headers:
I0905 09:35:45.240265    1515 round_trippers.go:447]     Date: Thu, 05 Sep 2019 09:35:45 GMT
I0905 09:35:45.240283    1515 round_trippers.go:447]     Content-Type: application/json
I0905 09:35:45.240346    1515 round_trippers.go:447]     Vary: Accept-Encoding
I0905 09:35:45.240383    1515 round_trippers.go:447]     Strict-Transport-Security: max-age=15724800; includeSubDomains
I0905 09:35:45.240398    1515 round_trippers.go:447]     Server: nginx/1.15.6
I0905 09:35:45.240803    1515 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"example-ruby-api-95f9c4d67-hx6ls","generateName":"example-ruby-api-95f9c4d67-","namespace":"foo-development-early-adopters","selfLink":"/api/v1/namespaces/foo-development-early-adopters/pods/example-ruby-api-95f9c4d67-hx6ls","uid":"32414712-cfc0-11e9-909a-08002719ef83","resourceVersion":"9746","creationTimestamp":"2019-09-05T09:33:19Z","labels":{"app.kubernetes.io/instance":"example-ruby-api","app.kubernetes.io/name":"example-ruby-api","pod-template-hash":"95f9c4d67"},"annotations":{"cni.projectcalico.org/podIP":"10.42.2.8/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"example-ruby-api-95f9c4d67","uid":"3239ae39-cfc0-11e9-909a-08002719ef83","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"config","configMap":{"name":"example-ruby-api","defaultMode":420}},{"name":"secret","secret":{"secretName":"example-ruby-api","defaultMode":420}},{"name":"default-token-dt2v2","secret":{"secretName":"default-token-dt2v2","defaultMode":420}}],"containers":[{"name":"example-ruby-api","image":"quay.io/mycompany/example-ruby-api:e088063bff4bd3fdd6005278d820f730f3f04307","ports":[{"name":"http","containerPort":3000,"protocol":"TCP"}],"env":[{"name":"APP_STAGE","value":"development-early-adopters"}],"resources":{},"volumeMounts":[{"name":"config","mountPath":"/etc/app/config"},{"name":"secret","mountPath":"/etc/app/secret"},{"name":"default-token-dt2v2","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/status","port":3000,"scheme":"HTTP"},"initialDelaySeconds":15,"timeoutSeconds":3,"periodSeconds":10,"successThreshold":1,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/status","port":3000,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":10},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"k8s-node-3","securityContext":{},"imagePullSecrets":[{"name":"quay-mycompany"}],"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-09-05T09:33:19Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-09-05T09:33:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-09-05T09:33:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-09-05T09:33:19Z"}],"hostIP":"10.0.2.15","podIP":"10.42.2.8","startTime":"2019-09-05T09:33:19Z","containerStatuses":[{"name":"example-ruby-api","state":{"running":{"startedAt":"2019-09-05T09:33:25Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"quay.io/mycompany/example-ruby-api:e088063bff4bd3fdd6005278d820f730f3f04307","imageID":"docker-pullable://quay.io/mycompany/example-ruby-api@sha256:e7fbf0797b6f94b1017eb7ac00d165389c8c63f71a9c274ffa080a0a5f9045af","containerID":"docker://6ddc8418b97c1e0c8a2c05e1dc75ec6272c85ab383e3cebf0167368390f3df78"}],"qosClass":"BestEffort"}}
I0905 09:35:45.245674    1515 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.2 (linux/amd64) kubernetes/f627830" -H "Authorization: Bearer kubeconfig-user-66lw9.c-sl8lh:cwprntbbt54g7rdqghgl98qknkr69p54qzv5wbp2c5zhvp2jdjgbnr" 'https://rancher.localdev/k8s/clusters/c-sl8lh/api/v1/namespaces/foo-development-early-adopters/pods/example-ruby-api-95f9c4d67-hx6ls/log'
I0905 09:35:45.257575    1515 round_trippers.go:438] GET https://rancher.localdev/k8s/clusters/c-sl8lh/api/v1/namespaces/foo-development-early-adopters/pods/example-ruby-api-95f9c4d67-hx6ls/log 404 Not Found in 11 milliseconds
I0905 09:35:45.257627    1515 round_trippers.go:444] Response Headers:
I0905 09:35:45.257648    1515 round_trippers.go:447]     Date: Thu, 05 Sep 2019 09:35:45 GMT
I0905 09:35:45.257668    1515 round_trippers.go:447]     Content-Type: application/json
I0905 09:35:45.257708    1515 round_trippers.go:447]     Vary: Accept-Encoding
I0905 09:35:45.257750    1515 round_trippers.go:447]     Strict-Transport-Security: max-age=15724800; includeSubDomains
I0905 09:35:45.257920    1515 round_trippers.go:447]     Server: nginx/1.15.6
I0905 09:35:45.258046    1515 request.go:947] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource ( pods/log example-ruby-api-95f9c4d67-hx6ls)","reason":"NotFound","details":{"name":"example-ruby-api-95f9c4d67-hx6ls","kind":"pods/log"},"code":404}
I0905 09:35:45.258608    1515 helpers.go:199] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server could not find the requested resource ( pods/log example-ruby-api-95f9c4d67-hx6ls)",
  "reason": "NotFound",
  "details": {
    "name": "example-ruby-api-95f9c4d67-hx6ls",
    "kind": "pods/log"
  },
  "code": 404
}]
F0905 09:35:45.258665    1515 helpers.go:114] Error from server (NotFound): the server could not find the requested resource ( pods/log example-ruby-api-95f9c4d67-hx6ls)

kubectl exec -it example-ruby-api-95f9c4d67-hx6ls bash

error: unable to upgrade connection: unable to read error from server response

kubectl get nodes -o json

{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "Node",
            "metadata": {
                "annotations": {
                    "flannel.alpha.coreos.com/backend-data": "{\"VtepMAC\":\"42:5d:d0:3d:d8:4c\"}",
                    "flannel.alpha.coreos.com/backend-type": "vxlan",
                    "flannel.alpha.coreos.com/kube-subnet-manager": "true",
                    "flannel.alpha.coreos.com/public-ip": "192.168.99.110",
                    "node.alpha.kubernetes.io/ttl": "0",
                    "rke.cattle.io/external-ip": "192.168.99.110",
                    "rke.cattle.io/internal-ip": "192.168.99.110",
                    "volumes.kubernetes.io/controller-managed-attach-detach": "true"
                },
                "creationTimestamp": "2019-09-05T08:21:55Z",
                "labels": {
                    "beta.kubernetes.io/arch": "amd64",
                    "beta.kubernetes.io/os": "linux",
                    "kubernetes.io/arch": "amd64",
                    "kubernetes.io/hostname": "k8s-node-1",
                    "kubernetes.io/os": "linux",
                    "node-role.kubernetes.io/controlplane": "true",
                    "node-role.kubernetes.io/etcd": "true",
                    "node-role.kubernetes.io/worker": "true"
                },
                "name": "k8s-node-1",
                "resourceVersion": "9083",
                "selfLink": "/api/v1/nodes/k8s-node-1",
                "uid": "38fb829d-cfb6-11e9-9b19-0800277f5a92"
            },
            "spec": {
                "podCIDR": "10.42.0.0/24"
            },
            "status": {
                "addresses": [
                    {
                        "address": "10.0.2.15",
                        "type": "InternalIP"
                    },
                    {
                        "address": "k8s-node-1",
                        "type": "Hostname"
                    }
                ],
                "allocatable": {
                    "cpu": "1",
                    "ephemeral-storage": "18457082236",
                    "hugepages-2Mi": "0",
                    "memory": "1941752Ki",
                    "pods": "110"
                },
                "capacity": {
                    "cpu": "1",
                    "ephemeral-storage": "20027216Ki",
                    "hugepages-2Mi": "0",
                    "memory": "2044152Ki",
                    "pods": "110"
                },
                "conditions": [
                    {
                        "lastHeartbeatTime": "2019-09-05T09:28:05Z",
                        "lastTransitionTime": "2019-09-05T08:21:55Z",
                        "message": "kubelet has sufficient memory available",
                        "reason": "KubeletHasSufficientMemory",
                        "status": "False",
                        "type": "MemoryPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:28:05Z",
                        "lastTransitionTime": "2019-09-05T08:21:55Z",
                        "message": "kubelet has no disk pressure",
                        "reason": "KubeletHasNoDiskPressure",
                        "status": "False",
                        "type": "DiskPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:28:05Z",
                        "lastTransitionTime": "2019-09-05T08:21:55Z",
                        "message": "kubelet has sufficient PID available",
                        "reason": "KubeletHasSufficientPID",
                        "status": "False",
                        "type": "PIDPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:28:05Z",
                        "lastTransitionTime": "2019-09-05T08:22:26Z",
                        "message": "kubelet is posting ready status",
                        "reason": "KubeletReady",
                        "status": "True",
                        "type": "Ready"
                    }
                ],
                "daemonEndpoints": {
                    "kubeletEndpoint": {
                        "Port": 10250
                    }
                },
                "images": [
                    {
                        "names": [
                            "rancher/hyperkube@sha256:470cf377ee28909133c2b1d881bbba516b59a5a9ee3e032a14769a5aed75df8d",
                            "rancher/hyperkube:v1.14.6-rancher1"
                        ],
                        "sizeBytes": 1051460949
                    },
                    {
                        "names": [
                            "rancher/nginx-ingress-controller@sha256:5a60a68777ed1aa5472b987d1f085a653ada50056b7e6a43d22361973a24f2c0",
                            "rancher/nginx-ingress-controller:0.21.0-rancher3"
                        ],
                        "sizeBytes": 568519792
                    },
                    {
                        "names": [
                            "rancher/rancher-agent@sha256:6c1e84e208b87912683d1c68c80476d48150480a7f8819e8931c063492621d70",
                            "rancher/rancher-agent:v2.2.8"
                        ],
                        "sizeBytes": 285055691
                    },
                    {
                        "names": [
                            "rancher/kube-api-auth@sha256:3e666c1d288982daa3173ef53b02acda2b1f1d447b0c6eb3148b9a6df40ae217",
                            "rancher/kube-api-auth:v0.1.3"
                        ],
                        "sizeBytes": 148911098
                    },
                    {
                        "names": [
                            "rancher/rke-tools@sha256:b97b789d53eb42786c59d9910b893d9489933682f1d989fe5071d301a07041b1",
                            "rancher/rke-tools:v0.1.42"
                        ],
                        "sizeBytes": 131111312
                    },
                    {
                        "names": [
                            "rancher/calico-node@sha256:2f46157483904649f334da83e6b5e522bf20cbd3ea6ec9f43456673933021d5a",
                            "rancher/calico-node:v3.4.0"
                        ],
                        "sizeBytes": 75898396
                    },
                    {
                        "names": [
                            "rancher/calico-cni@sha256:b3f4045680586133039f1e77f3ebb4bb5e6c60fa85f184f395eee32303462715",
                            "rancher/calico-cni:v3.4.0"
                        ],
                        "sizeBytes": 75430420
                    },
                    {
                        "names": [
                            "rancher/cluster-proportional-autoscaler@sha256:51d5ecdd6b9067d89c911c08d8f11f35df6aba5241d1bb350b5c97538178e720",
                            "rancher/cluster-proportional-autoscaler:1.3.0"
                        ],
                        "sizeBytes": 45844959
                    },
                    {
                        "names": [
                            "rancher/coreos-flannel@sha256:57f85f0bb7360f2dd378d74ffc01ac99a4c152ad463455ff429867c6879f4386",
                            "rancher/coreos-flannel:v0.10.0"
                        ],
                        "sizeBytes": 44577768
                    },
                    {
                        "names": [
                            "rancher/metrics-server@sha256:33f1c8b87c9acc5ce53bbf816c8657627baa7af57e6ad07a30159e819a0912a6",
                            "rancher/metrics-server:v0.3.1"
                        ],
                        "sizeBytes": 40767713
                    },
                    {
                        "names": [
                            "rancher/coredns-coredns@sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b",
                            "rancher/coredns-coredns:1.3.1"
                        ],
                        "sizeBytes": 40283416
                    },
                    {
                        "names": [
                            "rancher/coreos-etcd@sha256:b2100f7b0dbd592e6e81689ac607055ed80f0bdbde7e29d24b4decf4cca63ea9",
                            "rancher/coreos-etcd:v3.3.10-rancher1"
                        ],
                        "sizeBytes": 39464037
                    },
                    {
                        "names": [
                            "rancher/nginx-ingress-controller-defaultbackend@sha256:010b50980bbabb8b174fd3bb5c2399cd91dfca01d16362f7a916e696149676d0",
                            "rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1"
                        ],
                        "sizeBytes": 5132544
                    },
                    {
                        "names": [
                            "rancher/pause@sha256:d22591b61e9c2b52aecbf07106d5db313c4f178e404d660b32517b18fcbf0144",
                            "rancher/pause:3.1"
                        ],
                        "sizeBytes": 742472
                    }
                ],
                "nodeInfo": {
                    "architecture": "amd64",
                    "bootID": "b161db0d-6c20-4512-a9f8-b728a3f84b3a",
                    "containerRuntimeVersion": "docker://18.9.8",
                    "kernelVersion": "4.14.138-rancher",
                    "kubeProxyVersion": "v1.14.6",
                    "kubeletVersion": "v1.14.6",
                    "machineID": "311936e2303b034fe7ef70182235b8cb",
                    "operatingSystem": "linux",
                    "osImage": "RancherOS v1.5.4",
                    "systemUUID": "E2A1C80B-422B-4A09-B959-0C5E51843322"
                }
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Node",
            "metadata": {
                "annotations": {
                    "flannel.alpha.coreos.com/backend-data": "{\"VtepMAC\":\"b2:5c:99:2a:11:4d\"}",
                    "flannel.alpha.coreos.com/backend-type": "vxlan",
                    "flannel.alpha.coreos.com/kube-subnet-manager": "true",
                    "flannel.alpha.coreos.com/public-ip": "192.168.99.111",
                    "node.alpha.kubernetes.io/ttl": "0",
                    "rke.cattle.io/external-ip": "192.168.99.111",
                    "rke.cattle.io/internal-ip": "192.168.99.111",
                    "volumes.kubernetes.io/controller-managed-attach-detach": "true"
                },
                "creationTimestamp": "2019-09-05T08:25:52Z",
                "labels": {
                    "beta.kubernetes.io/arch": "amd64",
                    "beta.kubernetes.io/os": "linux",
                    "kubernetes.io/arch": "amd64",
                    "kubernetes.io/hostname": "k8s-node-2",
                    "kubernetes.io/os": "linux",
                    "node-role.kubernetes.io/controlplane": "true",
                    "node-role.kubernetes.io/etcd": "true",
                    "node-role.kubernetes.io/worker": "true"
                },
                "name": "k8s-node-2",
                "resourceVersion": "9065",
                "selfLink": "/api/v1/nodes/k8s-node-2",
                "uid": "c5f5c685-cfb6-11e9-909a-08002719ef83"
            },
            "spec": {
                "podCIDR": "10.42.1.0/24"
            },
            "status": {
                "addresses": [
                    {
                        "address": "10.0.2.15",
                        "type": "InternalIP"
                    },
                    {
                        "address": "k8s-node-2",
                        "type": "Hostname"
                    }
                ],
                "allocatable": {
                    "cpu": "1",
                    "ephemeral-storage": "18457082236",
                    "hugepages-2Mi": "0",
                    "memory": "1941752Ki",
                    "pods": "110"
                },
                "capacity": {
                    "cpu": "1",
                    "ephemeral-storage": "20027216Ki",
                    "hugepages-2Mi": "0",
                    "memory": "2044152Ki",
                    "pods": "110"
                },
                "conditions": [
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:55Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has sufficient memory available",
                        "reason": "KubeletHasSufficientMemory",
                        "status": "False",
                        "type": "MemoryPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:55Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has no disk pressure",
                        "reason": "KubeletHasNoDiskPressure",
                        "status": "False",
                        "type": "DiskPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:55Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has sufficient PID available",
                        "reason": "KubeletHasSufficientPID",
                        "status": "False",
                        "type": "PIDPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:55Z",
                        "lastTransitionTime": "2019-09-05T08:30:42Z",
                        "message": "kubelet is posting ready status",
                        "reason": "KubeletReady",
                        "status": "True",
                        "type": "Ready"
                    }
                ],
                "daemonEndpoints": {
                    "kubeletEndpoint": {
                        "Port": 10250
                    }
                },
                "images": [
                    {
                        "names": [
                            "rancher/hyperkube@sha256:470cf377ee28909133c2b1d881bbba516b59a5a9ee3e032a14769a5aed75df8d",
                            "rancher/hyperkube:v1.14.6-rancher1"
                        ],
                        "sizeBytes": 1051460949
                    },
                    {
                        "names": [
                            "rancher/nginx-ingress-controller@sha256:5a60a68777ed1aa5472b987d1f085a653ada50056b7e6a43d22361973a24f2c0",
                            "rancher/nginx-ingress-controller:0.21.0-rancher3"
                        ],
                        "sizeBytes": 568519792
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:c9ba17177315c1d2d49b24cdc5591e8402c9d24fe1f12e161e5422b0b7256380",
                            "quay.io/mycompany/example-ruby-api:8ea432f968b7794ed0f0f46cd7b2a517a8ef5877"
                        ],
                        "sizeBytes": 351756701
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:4a3ff350883cc945d19a6a30bb424786db242c719b14cb8ee6f2007e927294f9",
                            "quay.io/mycompany/example-ruby-api:8a686df99962b826834beef2daf3de074991785c"
                        ],
                        "sizeBytes": 351756695
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:9aa7a4e2c9345292a9c817e5d57c4c110526e4595b0bd1097ebf9822be4c3419",
                            "quay.io/mycompany/example-ruby-api:476c4ac4598270f77cc0c3aec6e323dafa23cfe8"
                        ],
                        "sizeBytes": 351756632
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:2d3c4aee1f2f239dca8e0053275989ba903d5ee57d30c9184e1ca6c7a09063ef",
                            "quay.io/mycompany/example-ruby-api:ee816449691c8dc0b7b69abe3414353876c45841"
                        ],
                        "sizeBytes": 351756626
                    },
                    {
                        "names": [
                            "rancher/rancher-agent@sha256:6c1e84e208b87912683d1c68c80476d48150480a7f8819e8931c063492621d70",
                            "rancher/rancher-agent:v2.2.8"
                        ],
                        "sizeBytes": 285055691
                    },
                    {
                        "names": [
                            "rancher/kube-api-auth@sha256:3e666c1d288982daa3173ef53b02acda2b1f1d447b0c6eb3148b9a6df40ae217",
                            "rancher/kube-api-auth:v0.1.3"
                        ],
                        "sizeBytes": 148911098
                    },
                    {
                        "names": [
                            "rancher/rke-tools@sha256:b97b789d53eb42786c59d9910b893d9489933682f1d989fe5071d301a07041b1",
                            "rancher/rke-tools:v0.1.42"
                        ],
                        "sizeBytes": 131111312
                    },
                    {
                        "names": [
                            "rancher/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747",
                            "rancher/kubernetes-dashboard-amd64:v1.10.1"
                        ],
                        "sizeBytes": 121711221
                    },
                    {
                        "names": [
                            "rancher/calico-node@sha256:2f46157483904649f334da83e6b5e522bf20cbd3ea6ec9f43456673933021d5a",
                            "rancher/calico-node:v3.4.0"
                        ],
                        "sizeBytes": 75898396
                    },
                    {
                        "names": [
                            "rancher/calico-cni@sha256:b3f4045680586133039f1e77f3ebb4bb5e6c60fa85f184f395eee32303462715",
                            "rancher/calico-cni:v3.4.0"
                        ],
                        "sizeBytes": 75430420
                    },
                    {
                        "names": [
                            "rancher/coreos-flannel@sha256:57f85f0bb7360f2dd378d74ffc01ac99a4c152ad463455ff429867c6879f4386",
                            "rancher/coreos-flannel:v0.10.0"
                        ],
                        "sizeBytes": 44577768
                    },
                    {
                        "names": [
                            "rancher/coreos-etcd@sha256:b2100f7b0dbd592e6e81689ac607055ed80f0bdbde7e29d24b4decf4cca63ea9",
                            "rancher/coreos-etcd:v3.3.10-rancher1"
                        ],
                        "sizeBytes": 39464037
                    },
                    {
                        "names": [
                            "rancher/pause@sha256:d22591b61e9c2b52aecbf07106d5db313c4f178e404d660b32517b18fcbf0144",
                            "rancher/pause:3.1"
                        ],
                        "sizeBytes": 742472
                    }
                ],
                "nodeInfo": {
                    "architecture": "amd64",
                    "bootID": "ae41f273-50b8-4abb-9900-4b3094b5d32e",
                    "containerRuntimeVersion": "docker://18.9.8",
                    "kernelVersion": "4.14.138-rancher",
                    "kubeProxyVersion": "v1.14.6",
                    "kubeletVersion": "v1.14.6",
                    "machineID": "311936e2303b034fe7ef70182235b8cb",
                    "operatingSystem": "linux",
                    "osImage": "RancherOS v1.5.4",
                    "systemUUID": "D372D785-6E85-4B79-B354-EA3704A6BF47"
                }
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Node",
            "metadata": {
                "annotations": {
                    "flannel.alpha.coreos.com/backend-data": "{\"VtepMAC\":\"e6:89:58:22:8f:4a\"}",
                    "flannel.alpha.coreos.com/backend-type": "vxlan",
                    "flannel.alpha.coreos.com/kube-subnet-manager": "true",
                    "flannel.alpha.coreos.com/public-ip": "192.168.99.112",
                    "node.alpha.kubernetes.io/ttl": "0",
                    "rke.cattle.io/external-ip": "192.168.99.112",
                    "rke.cattle.io/internal-ip": "192.168.99.112",
                    "volumes.kubernetes.io/controller-managed-attach-detach": "true"
                },
                "creationTimestamp": "2019-09-05T08:25:52Z",
                "labels": {
                    "beta.kubernetes.io/arch": "amd64",
                    "beta.kubernetes.io/os": "linux",
                    "kubernetes.io/arch": "amd64",
                    "kubernetes.io/hostname": "k8s-node-3",
                    "kubernetes.io/os": "linux",
                    "node-role.kubernetes.io/controlplane": "true",
                    "node-role.kubernetes.io/etcd": "true",
                    "node-role.kubernetes.io/worker": "true"
                },
                "name": "k8s-node-3",
                "resourceVersion": "9064",
                "selfLink": "/api/v1/nodes/k8s-node-3",
                "uid": "c60936cb-cfb6-11e9-b9dc-08002747e680"
            },
            "spec": {
                "podCIDR": "10.42.2.0/24"
            },
            "status": {
                "addresses": [
                    {
                        "address": "10.0.2.15",
                        "type": "InternalIP"
                    },
                    {
                        "address": "k8s-node-3",
                        "type": "Hostname"
                    }
                ],
                "allocatable": {
                    "cpu": "1",
                    "ephemeral-storage": "18457082236",
                    "hugepages-2Mi": "0",
                    "memory": "1941752Ki",
                    "pods": "110"
                },
                "capacity": {
                    "cpu": "1",
                    "ephemeral-storage": "20027216Ki",
                    "hugepages-2Mi": "0",
                    "memory": "2044152Ki",
                    "pods": "110"
                },
                "conditions": [
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:54Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has sufficient memory available",
                        "reason": "KubeletHasSufficientMemory",
                        "status": "False",
                        "type": "MemoryPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:54Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has no disk pressure",
                        "reason": "KubeletHasNoDiskPressure",
                        "status": "False",
                        "type": "DiskPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:54Z",
                        "lastTransitionTime": "2019-09-05T08:25:52Z",
                        "message": "kubelet has sufficient PID available",
                        "reason": "KubeletHasSufficientPID",
                        "status": "False",
                        "type": "PIDPressure"
                    },
                    {
                        "lastHeartbeatTime": "2019-09-05T09:27:54Z",
                        "lastTransitionTime": "2019-09-05T08:30:32Z",
                        "message": "kubelet is posting ready status",
                        "reason": "KubeletReady",
                        "status": "True",
                        "type": "Ready"
                    }
                ],
                "daemonEndpoints": {
                    "kubeletEndpoint": {
                        "Port": 10250
                    }
                },
                "images": [
                    {
                        "names": [
                            "rancher/hyperkube@sha256:470cf377ee28909133c2b1d881bbba516b59a5a9ee3e032a14769a5aed75df8d",
                            "rancher/hyperkube:v1.14.6-rancher1"
                        ],
                        "sizeBytes": 1051460949
                    },
                    {
                        "names": [
                            "rancher/nginx-ingress-controller@sha256:5a60a68777ed1aa5472b987d1f085a653ada50056b7e6a43d22361973a24f2c0",
                            "rancher/nginx-ingress-controller:0.21.0-rancher3"
                        ],
                        "sizeBytes": 568519792
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:c9ba17177315c1d2d49b24cdc5591e8402c9d24fe1f12e161e5422b0b7256380",
                            "quay.io/mycompany/example-ruby-api:8ea432f968b7794ed0f0f46cd7b2a517a8ef5877"
                        ],
                        "sizeBytes": 351756701
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:4a3ff350883cc945d19a6a30bb424786db242c719b14cb8ee6f2007e927294f9",
                            "quay.io/mycompany/example-ruby-api:8a686df99962b826834beef2daf3de074991785c"
                        ],
                        "sizeBytes": 351756695
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:9aa7a4e2c9345292a9c817e5d57c4c110526e4595b0bd1097ebf9822be4c3419",
                            "quay.io/mycompany/example-ruby-api:476c4ac4598270f77cc0c3aec6e323dafa23cfe8"
                        ],
                        "sizeBytes": 351756632
                    },
                    {
                        "names": [
                            "quay.io/mycompany/example-ruby-api@sha256:2d3c4aee1f2f239dca8e0053275989ba903d5ee57d30c9184e1ca6c7a09063ef",
                            "quay.io/mycompany/example-ruby-api:ee816449691c8dc0b7b69abe3414353876c45841"
                        ],
                        "sizeBytes": 351756626
                    },
                    {
                        "names": [
                            "rancher/rancher-agent@sha256:6c1e84e208b87912683d1c68c80476d48150480a7f8819e8931c063492621d70",
                            "rancher/rancher-agent:v2.2.8"
                        ],
                        "sizeBytes": 285055691
                    },
                    {
                        "names": [
                            "rancher/kube-api-auth@sha256:3e666c1d288982daa3173ef53b02acda2b1f1d447b0c6eb3148b9a6df40ae217",
                            "rancher/kube-api-auth:v0.1.3"
                        ],
                        "sizeBytes": 148911098
                    },
                    {
                        "names": [
                            "rancher/rke-tools@sha256:b97b789d53eb42786c59d9910b893d9489933682f1d989fe5071d301a07041b1",
                            "rancher/rke-tools:v0.1.42"
                        ],
                        "sizeBytes": 131111312
                    },
                    {
                        "names": [
                            "rancher/calico-node@sha256:2f46157483904649f334da83e6b5e522bf20cbd3ea6ec9f43456673933021d5a",
                            "rancher/calico-node:v3.4.0"
                        ],
                        "sizeBytes": 75898396
                    },
                    {
                        "names": [
                            "rancher/calico-cni@sha256:b3f4045680586133039f1e77f3ebb4bb5e6c60fa85f184f395eee32303462715",
                            "rancher/calico-cni:v3.4.0"
                        ],
                        "sizeBytes": 75430420
                    },
                    {
                        "names": [
                            "rancher/coreos-flannel@sha256:57f85f0bb7360f2dd378d74ffc01ac99a4c152ad463455ff429867c6879f4386",
                            "rancher/coreos-flannel:v0.10.0"
                        ],
                        "sizeBytes": 44577768
                    },
                    {
                        "names": [
                            "rancher/coreos-etcd@sha256:b2100f7b0dbd592e6e81689ac607055ed80f0bdbde7e29d24b4decf4cca63ea9",
                            "rancher/coreos-etcd:v3.3.10-rancher1"
                        ],
                        "sizeBytes": 39464037
                    },
                    {
                        "names": [
                            "rancher/pause@sha256:d22591b61e9c2b52aecbf07106d5db313c4f178e404d660b32517b18fcbf0144",
                            "rancher/pause:3.1"
                        ],
                        "sizeBytes": 742472
                    }
                ],
                "nodeInfo": {
                    "architecture": "amd64",
                    "bootID": "41854b5f-87aa-46b9-9c8c-8556a510fbfd",
                    "containerRuntimeVersion": "docker://18.9.8",
                    "kernelVersion": "4.14.138-rancher",
                    "kubeProxyVersion": "v1.14.6",
                    "kubeletVersion": "v1.14.6",
                    "machineID": "311936e2303b034fe7ef70182235b8cb",
                    "operatingSystem": "linux",
                    "osImage": "RancherOS v1.5.4",
                    "systemUUID": "4D9B6AE9-FD4F-42D4-BD08-5C6C9C44EA52"
                }
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

docker logs kubelet (on the troubling node):

I0905 09:33:19.549779    4737 kubelet.go:1885] SyncLoop (ADD, "api"): "example-ruby-api-95f9c4d67-hx6ls_foo-development-early-adopters(32414712-cfc0-11e9-909a-08002719ef83)"
I0905 09:33:19.593045    4737 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-dt2v2" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-default-token-dt2v2") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.593223    4737 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config" (UniqueName: "kubernetes.io/configmap/32414712-cfc0-11e9-909a-08002719ef83-config") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.593297    4737 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "secret" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-secret") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.694328    4737 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-dt2v2" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-default-token-dt2v2") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.694500    4737 reconciler.go:252] operationExecutor.MountVolume started for volume "config" (UniqueName: "kubernetes.io/configmap/32414712-cfc0-11e9-909a-08002719ef83-config") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.694584    4737 reconciler.go:252] operationExecutor.MountVolume started for volume "secret" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-secret") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.699835    4737 operation_generator.go:669] MountVolume.SetUp succeeded for volume "default-token-dt2v2" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-default-token-dt2v2") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.700210    4737 operation_generator.go:669] MountVolume.SetUp succeeded for volume "config" (UniqueName: "kubernetes.io/configmap/32414712-cfc0-11e9-909a-08002719ef83-config") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.700553    4737 operation_generator.go:669] MountVolume.SetUp succeeded for volume "secret" (UniqueName: "kubernetes.io/secret/32414712-cfc0-11e9-909a-08002719ef83-secret") pod "example-ruby-api-95f9c4d67-hx6ls" (UID: "32414712-cfc0-11e9-909a-08002719ef83")
I0905 09:33:19.867281    4737 kuberuntime_manager.go:409] No sandbox for pod "example-ruby-api-95f9c4d67-hx6ls_foo-development-early-adopters(32414712-cfc0-11e9-909a-08002719ef83)" can be found. Need to start a new one
Calico CNI fetching podCidr from Kubernetes
Calico CNI passing podCidr to host-local IPAM: 10.42.2.0/24
Calico CNI using IPs: [10.42.2.8/32]
I0905 09:33:20.157381    4737 kubelet.go:1892] SyncLoop (UPDATE, "api"): "example-ruby-api-95f9c4d67-hx6ls_foo-development-early-adopters(32414712-cfc0-11e9-909a-08002719ef83)"
I0905 09:33:20.192131    4737 provider.go:124] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0905 09:33:20.207101    4737 kubelet.go:1930] SyncLoop (PLEG): "example-ruby-api-95f9c4d67-hx6ls_foo-development-early-adopters(32414712-cfc0-11e9-909a-08002719ef83)", event: &pleg.PodLifecycleEvent{ID:"32414712-cfc0-11e9-909a-08002719ef83", Type:"ContainerStarted", Data:"32054803638d46d2ca3dac720f260620692e7969511819b532f0d0261c07a021"}
I0905 09:33:25.281852    4737 kube_docker_client.go:348] Stop pulling image "quay.io/mycompany/example-ruby-api:e088063bff4bd3fdd6005278d820f730f3f04307": "Status: Downloaded newer image for quay.io/mycompany/example-ruby-api:e088063bff4bd3fdd6005278d820f730f3f04307"
I0905 09:33:26.256870    4737 kubelet.go:1930] SyncLoop (PLEG): "example-ruby-api-95f9c4d67-hx6ls_foo-development-early-adopters(32414712-cfc0-11e9-909a-08002719ef83)", event: &pleg.PodLifecycleEvent{ID:"32414712-cfc0-11e9-909a-08002719ef83", Type:"ContainerStarted", Data:"6ddc8418b97c1e0c8a2c05e1dc75ec6272c85ab383e3cebf0167368390f3df78"}
@pietervogelaar

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

I used kubectl (v1.15.2) on my local machine with the kubeconfig downloaded from Rancher UI. If I click on "Launch kubectl" in the UI, I get a 404 on GET wss://rancher.localdev/v3/clusters/c-sl8lh?shell=true. Message in the Chrome developer toolbar "Error during WebSocket handshake: Unexpected response code: 404".

@pietervogelaar

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

docker logs kube-apiserver (on troubling node)

+ grep -q cloud-provider=azure
+ echo kube-apiserver --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-prefix=/registry --service-node-port-range=30000-32767 --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --bind-address=0.0.0.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-allowed-names=kube-apiserver-proxy-client --requestheader-extra-headers-prefix=X-Remote-Extra- --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --anonymous-auth=false --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cloud-provider= --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --runtime-config=authorization.k8s.io/v1beta1=true --service-cluster-ip-range=10.43.0.0/16 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --advertise-address=192.168.99.112 --profiling=false --secure-port=6443 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --allow-privileged=true --insecure-port=0 --requestheader-username-headers=X-Remote-User --service-account-lookup=true --etcd-servers=https://192.168.99.112:2379,https://192.168.99.111:2379,https://192.168.99.110:2379 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem
+ '[' kube-apiserver = kubelet ']'
+ exec kube-apiserver --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-prefix=/registry --service-node-port-range=30000-32767 --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --bind-address=0.0.0.0 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-allowed-names=kube-apiserver-proxy-client --requestheader-extra-headers-prefix=X-Remote-Extra- --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authorization-mode=Node,RBAC --anonymous-auth=false --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cloud-provider= --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --runtime-config=authorization.k8s.io/v1beta1=true --service-cluster-ip-range=10.43.0.0/16 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --advertise-address=192.168.99.112 --profiling=false --secure-port=6443 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --allow-privileged=true --insecure-port=0 --requestheader-username-headers=X-Remote-User --service-account-lookup=true --etcd-servers=https://192.168.99.112:2379,https://192.168.99.111:2379,https://192.168.99.110:2379 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0905 08:25:23.439211       1 server.go:559] external host was not specified, using 192.168.99.112
I0905 08:25:23.439352       1 server.go:146] Version: v1.14.6
W0905 08:25:24.062754       1 authentication.go:368] the webhook cache ttl of 5s is shorter than the overall cache ttl of 10s for successful token authentication attempts.
I0905 08:25:24.068535       1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0905 08:25:24.068669       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
E0905 08:25:24.070633       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:24.070811       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:24.070908       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:24.070972       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:24.071032       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:24.071087       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I0905 08:25:24.071138       1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0905 08:25:24.071178       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0905 08:25:24.073231       1 client.go:352] parsed scheme: ""
I0905 08:25:24.073347       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.074017       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.074191       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.083782       1 client.go:352] parsed scheme: ""
I0905 08:25:24.083903       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.084286       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.084515       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:24.084613       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.085313       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.085613       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.103586       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.103864       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.104769       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.118198       1 master.go:233] Using reconciler: lease
I0905 08:25:24.119625       1 client.go:352] parsed scheme: ""
I0905 08:25:24.119700       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.119767       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.119870       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.131600       1 client.go:352] parsed scheme: ""
I0905 08:25:24.131731       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.131800       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.134078       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.134329       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.134464       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.134991       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.142863       1 client.go:352] parsed scheme: ""
I0905 08:25:24.142979       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.143043       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.143133       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:24.143300       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.149224       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.149802       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.157267       1 client.go:352] parsed scheme: ""
I0905 08:25:24.157618       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.157709       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.157867       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.158048       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.158183       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.158507       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.167060       1 client.go:352] parsed scheme: ""
I0905 08:25:24.167174       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.167238       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.167776       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.167969       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.168143       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.168406       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.181272       1 client.go:352] parsed scheme: ""
I0905 08:25:24.181428       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.181504       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.181642       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.182143       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.182416       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.182767       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.192744       1 client.go:352] parsed scheme: ""
I0905 08:25:24.192866       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.192932       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.193446       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.193635       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.193806       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.194576       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.210190       1 client.go:352] parsed scheme: ""
I0905 08:25:24.210301       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.210372       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.210451       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.210607       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.210766       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.211130       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.221009       1 client.go:352] parsed scheme: ""
I0905 08:25:24.221130       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.221195       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.223422       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.224121       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.224860       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.225929       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.239308       1 client.go:352] parsed scheme: ""
I0905 08:25:24.240726       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.241015       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.242753       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.243003       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.243236       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.243672       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.270337       1 client.go:352] parsed scheme: ""
I0905 08:25:24.270590       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.270721       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.271032       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.271224       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.271404       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.272360       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.286166       1 client.go:352] parsed scheme: ""
I0905 08:25:24.286287       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.286352       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.286664       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.286858       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.287023       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.287332       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.303257       1 client.go:352] parsed scheme: ""
I0905 08:25:24.303414       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.303484       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.306613       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.306840       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.307142       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.307569       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.316515       1 client.go:352] parsed scheme: ""
I0905 08:25:24.316743       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.316960       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.319624       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.319855       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.320040       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.320555       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.336632       1 client.go:352] parsed scheme: ""
I0905 08:25:24.336844       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.336909       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.336988       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
I0905 08:25:24.337178       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.337628       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.338202       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.349230       1 client.go:352] parsed scheme: ""
I0905 08:25:24.349346       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.349409       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.349491       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.349933       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.350209       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.350582       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.373579       1 client.go:352] parsed scheme: ""
I0905 08:25:24.373698       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.373762       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.373852       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.374047       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.374150       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.374415       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.389359       1 client.go:352] parsed scheme: ""
I0905 08:25:24.390315       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.390419       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.390829       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.391024       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.391106       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.391526       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.416838       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.417206       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.417851       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.539041       1 client.go:352] parsed scheme: ""
I0905 08:25:24.542498       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.547592       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.547818       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.574133       1 client.go:352] parsed scheme: ""
I0905 08:25:24.574218       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.574283       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.574368       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.575176       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.575507       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.575807       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.594974       1 client.go:352] parsed scheme: ""
I0905 08:25:24.596496       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.596638       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.596919       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.598347       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.598553       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.598840       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.608514       1 client.go:352] parsed scheme: ""
I0905 08:25:24.608622       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.608689       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.608884       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.608998       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.609149       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.609448       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.620983       1 client.go:352] parsed scheme: ""
I0905 08:25:24.621102       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.621172       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.621391       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.621589       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.621794       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.622189       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.631047       1 client.go:352] parsed scheme: ""
I0905 08:25:24.632240       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.632435       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.632673       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.632957       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.633257       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.634611       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.650624       1 client.go:352] parsed scheme: ""
I0905 08:25:24.650742       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.650809       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.651012       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.651205       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.651378       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.651638       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.662741       1 client.go:352] parsed scheme: ""
I0905 08:25:24.662853       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.662918       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.663109       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
I0905 08:25:24.663302       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.663556       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:24.664093       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.680335       1 client.go:352] parsed scheme: ""
I0905 08:25:24.680440       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.680495       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.681887       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.682515       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.682706       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.692880       1 client.go:352] parsed scheme: ""
I0905 08:25:24.693123       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.693198       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.693712       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.693986       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.694138       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.694622       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.710208       1 client.go:352] parsed scheme: ""
I0905 08:25:24.710326       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.710392       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.710553       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.710748       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.710932       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.711297       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.720361       1 client.go:352] parsed scheme: ""
I0905 08:25:24.720478       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.720538       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.720623       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.720793       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.720960       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.721265       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.736346       1 client.go:352] parsed scheme: ""
I0905 08:25:24.736530       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.736729       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.737076       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.737943       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.738306       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.739120       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.762261       1 client.go:352] parsed scheme: ""
I0905 08:25:24.762419       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.762491       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.762726       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.762927       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.763089       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.763360       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.776361       1 client.go:352] parsed scheme: ""
I0905 08:25:24.776470       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.776532       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.781182       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.110:2379 <nil>}]
I0905 08:25:24.781523       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.782418       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.793140       1 client.go:352] parsed scheme: ""
I0905 08:25:24.793257       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.793317       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.793393       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.793560       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.793730       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.794000       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.810048       1 client.go:352] parsed scheme: ""
I0905 08:25:24.810160       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.810221       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.810298       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.810479       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.810693       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.811005       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.825298       1 client.go:352] parsed scheme: ""
I0905 08:25:24.825698       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.825786       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.825893       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.826073       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.826339       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.826599       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.842207       1 client.go:352] parsed scheme: ""
I0905 08:25:24.842394       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.842576       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.842792       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:24.842899       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.843061       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.843640       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.864904       1 client.go:352] parsed scheme: ""
I0905 08:25:24.865222       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.865304       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.871330       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.110:2379 <nil>}]
I0905 08:25:24.872120       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.900469       1 client.go:352] parsed scheme: ""
I0905 08:25:24.900784       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.900871       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.901621       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.902865       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.903068       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:24.925677       1 client.go:352] parsed scheme: ""
I0905 08:25:24.925800       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.926416       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.927220       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:24.929471       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.934042       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:24.944887       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:24.997479       1 client.go:352] parsed scheme: ""
I0905 08:25:24.997663       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:24.997752       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:24.997922       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:24.998105       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.000776       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.001119       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.022958       1 client.go:352] parsed scheme: ""
I0905 08:25:25.023088       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.023169       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.023993       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.025188       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.025440       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.027278       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.041253       1 client.go:352] parsed scheme: ""
I0905 08:25:25.041527       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.042056       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.045840       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.075291       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.075444       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.076014       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.081469       1 client.go:352] parsed scheme: ""
I0905 08:25:25.082170       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.082622       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.083664       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:25.137665       1 client.go:352] parsed scheme: ""
I0905 08:25:25.137961       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.138168       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.138430       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
I0905 08:25:25.138542       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.138760       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.139070       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.140370       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.140547       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.140954       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.155379       1 client.go:352] parsed scheme: ""
I0905 08:25:25.155496       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.155562       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.155976       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.156177       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.158912       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.159481       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.169536       1 client.go:352] parsed scheme: ""
I0905 08:25:25.169658       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.169725       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.169814       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.170020       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.170190       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.170626       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.186483       1 client.go:352] parsed scheme: ""
I0905 08:25:25.188908       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.189148       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.189485       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.189961       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.190703       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.191542       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.205894       1 client.go:352] parsed scheme: ""
I0905 08:25:25.206006       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.206070       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.206165       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.206354       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.206667       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.207158       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.228667       1 client.go:352] parsed scheme: ""
I0905 08:25:25.228789       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.229026       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.229601       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:25.230144       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.230897       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:25.254942       1 client.go:352] parsed scheme: ""
I0905 08:25:25.255518       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.255794       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.255908       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:25.256095       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.257635       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.258052       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.272769       1 client.go:352] parsed scheme: ""
I0905 08:25:25.273977       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.275009       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.273805       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.275278       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.275354       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.275635       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.285609       1 client.go:352] parsed scheme: ""
I0905 08:25:25.285725       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.285800       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.285887       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.286193       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.286386       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.286837       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.298789       1 client.go:352] parsed scheme: ""
I0905 08:25:25.298911       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.298979       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.299068       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.299422       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.299603       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.300090       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.311058       1 client.go:352] parsed scheme: ""
I0905 08:25:25.311175       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.311243       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.311323       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.311508       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.311698       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.311991       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.348207       1 client.go:352] parsed scheme: ""
I0905 08:25:25.348333       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.348408       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.348487       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.348725       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.349063       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.349394       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.373089       1 client.go:352] parsed scheme: ""
I0905 08:25:25.373359       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.373452       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.373540       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
I0905 08:25:25.374115       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.374407       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.377421       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.398618       1 client.go:352] parsed scheme: ""
I0905 08:25:25.398757       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.398931       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.399031       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
I0905 08:25:25.399494       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.399987       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.406579       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.423731       1 client.go:352] parsed scheme: ""
I0905 08:25:25.426345       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.426603       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.426705       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.110:2379 <nil>}]
I0905 08:25:25.427216       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.427589       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.428620       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.439290       1 client.go:352] parsed scheme: ""
I0905 08:25:25.439414       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.439485       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.439780       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:25.439979       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.440298       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.440578       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.458213       1 client.go:352] parsed scheme: ""
I0905 08:25:25.469444       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.469961       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.459714       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.471659       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.471748       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.472306       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.483388       1 client.go:352] parsed scheme: ""
I0905 08:25:25.483515       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.483593       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.483903       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:25.484282       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.484655       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.484750       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.495794       1 client.go:352] parsed scheme: ""
I0905 08:25:25.496026       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.496107       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.496234       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:25.496667       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.497291       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.502660       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.511388       1 client.go:352] parsed scheme: ""
I0905 08:25:25.511653       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.511749       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.511847       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.512164       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.512357       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.512890       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.553630       1 client.go:352] parsed scheme: ""
I0905 08:25:25.553858       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.555135       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.559147       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:25.560450       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.561793       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.583198       1 client.go:352] parsed scheme: ""
I0905 08:25:25.583325       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.583447       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.583695       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.583935       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.584174       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.587519       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.620565       1 client.go:352] parsed scheme: ""
I0905 08:25:25.620706       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.620778       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.620869       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:25.621047       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.621337       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.622931       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.654458       1 client.go:352] parsed scheme: ""
I0905 08:25:25.654591       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.654786       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.655302       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.110:2379 <nil>}]
I0905 08:25:25.661000       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.661834       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.663299       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.686073       1 client.go:352] parsed scheme: ""
I0905 08:25:25.686362       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.686447       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.686560       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.686785       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.687147       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.688619       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.712738       1 client.go:352] parsed scheme: ""
I0905 08:25:25.717546       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.717906       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.718389       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
I0905 08:25:25.719090       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.726462       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.728004       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.756823       1 client.go:352] parsed scheme: ""
I0905 08:25:25.757080       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:25.757184       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:25.757723       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:25:25.757897       1 asm_amd64.s:1337] Failed to dial 192.168.99.112:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.758084       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
W0905 08:25:25.758569       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:25.795018       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:25.795572       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:25:25.796004       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
W0905 08:25:26.013948       1 genericapiserver.go:344] Skipping API batch/v2alpha1 because it has no resources.
W0905 08:25:26.021731       1 genericapiserver.go:344] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0905 08:25:26.024312       1 genericapiserver.go:344] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0905 08:25:26.024848       1 genericapiserver.go:344] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0905 08:25:26.026163       1 genericapiserver.go:344] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
E0905 08:25:27.203184       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:27.203410       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:27.203505       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:27.203588       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:27.203799       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0905 08:25:27.204071       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I0905 08:25:27.204182       1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0905 08:25:27.204236       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0905 08:25:27.206553       1 client.go:352] parsed scheme: ""
I0905 08:25:27.206681       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:27.206764       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:27.206858       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:27.238364       1 client.go:352] parsed scheme: ""
I0905 08:25:27.238515       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:25:27.238983       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:25:27.239419       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:27.240439       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:27.240824       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:25:27.269813       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:25:27.298150       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
W0905 08:25:27.313214       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
I0905 08:25:29.091437       1 secure_serving.go:116] Serving securely on [::]:6443
I0905 08:25:29.097022       1 controller.go:81] Starting OpenAPI AggregationController
I0905 08:25:29.097206       1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0905 08:25:29.097269       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0905 08:25:29.097328       1 autoregister_controller.go:139] Starting autoregister controller
I0905 08:25:29.097444       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0905 08:25:29.097929       1 crd_finalizer.go:242] Starting CRDFinalizer
I0905 08:25:29.098044       1 available_controller.go:320] Starting AvailableConditionController
I0905 08:25:29.098105       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0905 08:25:29.103457       1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0905 08:25:29.103587       1 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller
I0905 08:25:29.103647       1 customresource_discovery_controller.go:208] Starting DiscoveryController
I0905 08:25:29.103701       1 naming_controller.go:284] Starting NamingConditionController
I0905 08:25:29.103746       1 establishing_controller.go:73] Starting EstablishingController
E0905 08:25:29.163011       1 controller.go:148] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.99.112, ResourceVersion: 0, AdditionalErrorMsg:
I0905 08:25:29.199117       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0905 08:25:29.199794       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0905 08:25:29.212289       1 cache.go:39] Caches are synced for autoregister controller
I0905 08:25:29.238203       1 controller_utils.go:1034] Caches are synced for crd-autoregister controller
E0905 08:25:29.661082       1 autoregister_controller.go:193] v3.cluster.cattle.io failed with : apiservices.apiregistration.k8s.io "v3.cluster.cattle.io" already exists
E0905 08:25:29.681244       1 autoregister_controller.go:193] v1.crd.projectcalico.org failed with : apiservices.apiregistration.k8s.io "v1.crd.projectcalico.org" already exists
E0905 08:25:29.681483       1 autoregister_controller.go:193] v1.monitoring.coreos.com failed with : apiservices.apiregistration.k8s.io "v1.monitoring.coreos.com" already exists
I0905 08:25:30.090675       1 controller.go:107] OpenAPI AggregationController: Processing item
I0905 08:25:30.090831       1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0905 08:25:30.090963       1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0905 08:25:30.112421       1 storage_scheduling.go:122] all system priority classes are created successfully or already exist.
W0905 08:25:30.515103       1 lease.go:222] Resetting endpoints for master service "kubernetes" to [192.168.99.112]
I0905 08:25:30.522275       1 controller.go:606] quota admission added evaluator for: endpoints
I0905 08:25:30.933367       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:25:34.205898       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:25:39.217007       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:25:44.228250       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:25:49.228792       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:25:52.406883       1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
E0905 08:25:54.232715       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:25:59.239283       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:00.934369       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:26:00.934522       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:26:04.239855       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:26:05.257423       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:26:09.243530       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:26:14.248803       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:19.249259       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:24.252940       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:26:29.259113       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:34.259642       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:35.257947       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:26:35.258268       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:26:39.262812       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:26:44.268629       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:26:45.269264       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:26:49.269275       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:26:54.273013       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:26:59.279937       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:04.280377       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:09.284049       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:27:14.290220       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:15.269900       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:27:15.270058       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:27:19.291064       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:27:20.319148       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:27:24.294845       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:27:29.300525       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:34.304589       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:27:39.308113       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:27:44.313991       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:49.315059       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:27:50.320138       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:27:50.320443       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:27:54.318826       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:27:59.324674       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:28:00.324427       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:28:04.325323       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:09.328599       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:28:14.334846       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:19.335425       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:24.339249       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:28:29.344837       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:30.325554       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:28:30.325738       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:28:34.353742       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0905 08:28:35.382404       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:28:39.357140       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:28:44.362900       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:49.363586       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:28:54.367842       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:28:59.373741       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:29:04.377102       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:05.383398       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:29:05.383543       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:29:09.379964       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:14.385691       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:29:15.386815       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:29:19.386415       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:29:24.390118       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:29.395293       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:29:34.398331       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:39.401547       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:44.408221       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:29:45.387682       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:29:45.388070       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:29:49.409295       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:29:50.445377       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:29:54.412934       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:29:59.417707       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:04.420670       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:30:09.423931       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:30:14.432440       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:19.432788       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:20.446450       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:30:20.446468       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:30:24.437765       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0905 08:30:27.302380       1 client.go:352] parsed scheme: ""
I0905 08:30:27.302526       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:30:27.302586       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:30:27.302652       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:30:27.311738       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:30:27.312071       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:30:27.312424       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:30:27.315022       1 client.go:352] parsed scheme: ""
I0905 08:30:27.315104       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:30:27.315169       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:30:27.315780       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:30:27.323314       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:30:27.323642       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:30:27.323936       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
E0905 08:30:29.443442       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:30:30.202752       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:30:34.448603       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0905 08:30:38.677338       1 client.go:352] parsed scheme: ""
I0905 08:30:38.677465       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:30:38.677534       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:30:38.677611       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:30:38.687983       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:30:38.688404       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:30:38.693026       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:30:38.709996       1 client.go:352] parsed scheme: ""
I0905 08:30:38.710211       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:30:38.710283       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:30:38.710365       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:30:38.720942       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>}]
W0905 08:30:38.721313       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
W0905 08:30:38.721618       1 asm_amd64.s:1337] Failed to dial 192.168.99.111:2379: grpc: the connection is closing; please retry.
I0905 08:30:38.733197       1 client.go:352] parsed scheme: ""
I0905 08:30:38.733342       1 client.go:352] scheme "" not registered, fallback to default scheme
I0905 08:30:38.733429       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{192.168.99.112:2379 0  <nil>}]
I0905 08:30:38.733531       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.112:2379 <nil>} {192.168.99.111:2379 <nil>} {192.168.99.110:2379 <nil>}]
I0905 08:30:38.807918       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{192.168.99.111:2379 <nil>}]
W0905 08:30:38.809671       1 asm_amd64.s:1337] Failed to dial 192.168.99.110:2379: grpc: the connection is closing; please retry.
E0905 08:30:39.451425       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:30:44.459178       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:49.459956       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:54.501775       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:30:59.502101       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:31:00.204182       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:31:00.204294       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0905 08:31:04.508701       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0905 08:31:05.509838       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0905 08:31:09.510148       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:31:14.518403       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:31:19.518687       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:31:24.525698       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Get https://10.43.55.105:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0905 08:31:24.543563       1 available_controller.go:353] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0905 08:31:35.510550       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'dial tcp 10.43.55.105:443: i/o timeout'
Trying to reach: 'https://10.43.55.105:443/openapi/v2', Header: map[]
I0905 08:31:35.510707       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0905 08:32:35.511465       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:33:36.020613       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:34:36.023441       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:35:36.026279       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:36:36.038008       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:37:36.041733       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:38:36.044564       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:39:36.048011       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:40:36.051391       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:41:36.054551       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:42:36.057947       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:43:36.061072       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:44:36.064400       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:45:36.067658       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:46:36.070896       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:47:36.073929       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:48:36.077286       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:49:36.080383       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:50:36.083690       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:51:36.087290       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:52:36.091100       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:53:36.094385       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:54:36.097579       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:55:36.100774       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:56:36.104235       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:57:36.109496       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:58:36.112879       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 08:59:36.116134       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:00:36.119293       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:01:36.122813       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:02:36.126498       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:03:36.129797       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:04:36.132977       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:05:36.136758       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:06:36.139825       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:07:36.143004       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:08:36.146023       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:09:36.149215       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:10:36.152265       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:11:36.155940       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:12:36.159181       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:13:36.162788       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:14:36.166020       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:15:36.169289       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:16:36.174341       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:17:36.178068       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:18:36.181536       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:19:36.184860       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:20:36.193321       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:21:36.196959       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:22:36.200278       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:23:36.203552       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:24:36.206297       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:25:36.209275       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:26:36.212136       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:27:36.215192       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:28:36.218249       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:29:36.221841       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:30:36.224941       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:31:36.229112       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:32:36.234589       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:33:36.241760       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:34:36.245113       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:35:36.248335       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:36:36.251722       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:37:36.254712       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:38:36.257996       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:39:36.260971       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:40:36.264188       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:41:36.267599       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:42:36.270876       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:43:36.273641       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:44:36.276641       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0905 09:45:36.280487       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
@pietervogelaar

This comment has been minimized.

Copy link
Author

commented Sep 9, 2019

I fixed the problem. I also had to pass IP address as --node-name parameter to the agent join command:

--node-name 192.168.99.101 --address 192.168.99.101 --internal-address 192.168.99.101

Strange though that otherwise one node worked and the other one node not. Because both k8s-node-2 and k8s-node-3 DNS names are not resolvable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.