etcd crashing on upgraded 3.11 instance due to failing liveness probe #21609

mlbiam · 2018-12-04T15:40:02Z

After starting the master the api server dies after some time. it looks like the issue is related to etcd because the etcd pod continuosly crashes until it takes the api server out with it. It usually takes about 10 minutes. Looking in the etcd logs I see:

2018-12-04 15:27:30.544094 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://X.X.X.X:2379
2018-12-04 15:27:30.544551 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/server.crt
2018-12-04 15:27:30.544565 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2018-12-04 15:27:30.544581 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/
2018-12-04 15:27:30.544589 I | pkg/flags: recognized and used environment variable ETCD_DEBUG=False
2018-12-04 15:27:30.544602 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=2500
2018-12-04 15:27:30.544620 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=500
2018-12-04 15:27:30.544633 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.0.4.57:2380
2018-12-04 15:27:30.544648 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2018-12-04 15:27:30.544667 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
2018-12-04 15:27:30.544677 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/server.key
2018-12-04 15:27:30.544687 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://10.0.4.57:2379
2018-12-04 15:27:30.544696 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://10.0.4.57:2380
2018-12-04 15:27:30.544723 I | pkg/flags: recognized and used environment variable ETCD_NAME=ip-10-0-4-57.ec2.internal
2018-12-04 15:27:30.544734 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
2018-12-04 15:27:30.544742 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2018-12-04 15:27:30.544752 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
2018-12-04 15:27:30.544761 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-04 15:27:30.544778 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=4294967296
2018-12-04 15:27:30.544794 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-04 15:27:30.544828 W | pkg/flags: unrecognized environment variable ETCD_INITIAL_CLUSTER=
2018-12-04 15:27:30.544868 I | etcdmain: etcd Version: 3.2.22
2018-12-04 15:27:30.544876 I | etcdmain: Git SHA: 1674e682f
2018-12-04 15:27:30.544881 I | etcdmain: Go Version: go1.8.7
2018-12-04 15:27:30.544887 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-04 15:27:30.544892 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-04 15:27:30.544957 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180526200403 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544965 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180526200704 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544971 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180811182910 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544976 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20181126013050 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544981 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180526200357 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544990 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180526200658 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.544996 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180811182818 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.545001 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20181126012952 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
2018-12-04 15:27:30.545012 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-12-04 15:27:30.545043 I | embed: peerTLS: cert = /etc/etcd/peer.crt, key = /etc/etcd/peer.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-04 15:27:30.546220 I | embed: listening for peers on https://X.X.X.X:2380
2018-12-04 15:27:30.546300 I | embed: listening for client requests on X.X.X.X:2379
2018-12-04 15:27:30.561379 I | etcdserver: recovered store from snapshot at index 64300643
2018-12-04 15:27:30.564654 I | mvcc: restore compact to 57105449
2018-12-04 15:27:30.614314 I | etcdserver: name = ip-X-X-X-X
2018-12-04 15:27:30.614364 I | etcdserver: data dir = /var/lib/etcd/
2018-12-04 15:27:30.614387 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-04 15:27:30.614398 I | etcdserver: heartbeat = 500ms
2018-12-04 15:27:30.614404 I | etcdserver: election = 2500ms
2018-12-04 15:27:30.614410 I | etcdserver: snapshot count = 100000
2018-12-04 15:27:30.614437 I | etcdserver: advertise client URLs = https://X.X.X.X:2379
2018-12-04 15:27:30.769559 I | etcdserver: restarting member ccd4cb5684c5b1d4 in cluster d36b3a36535103f8 at commit index 64316408
2018-12-04 15:27:30.772214 I | raft: ccd4cb5684c5b1d4 became follower at term 109
2018-12-04 15:27:30.772263 I | raft: newRaft ccd4cb5684c5b1d4 [peers: [ccd4cb5684c5b1d4], term: 109, commit: 64316408, applied: 64300643, lastindex: 64316408, lastterm: 109]
2018-12-04 15:27:30.773451 I | etcdserver/api: enabled capabilities for version 3.2
2018-12-04 15:27:30.773487 I | etcdserver/membership: added member ccd4cb5684c5b1d4 [https://10.0.4.57:2380] to cluster d36b3a36535103f8 from store
2018-12-04 15:27:30.773500 I | etcdserver/membership: set the cluster version to 3.2 from store
2018-12-04 15:27:30.793965 I | mvcc: restore compact to 57105449
2018-12-04 15:27:30.827545 W | auth: simple token is not cryptographically signed
2018-12-04 15:27:30.833275 I | etcdserver: starting server... [version: 3.2.22, cluster version: 3.2]
2018-12-04 15:27:30.835797 I | embed: ClientTLS: cert = /etc/etcd/server.crt, key = /etc/etcd/server.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-04 15:27:30.837262 I | etcdserver: ccd4cb5684c5b1d4 as single-node; fast-forwarding 4 ticks (election ticks 5)
2018-12-04 15:27:31.773948 I | raft: ccd4cb5684c5b1d4 is starting a new election at term 109
2018-12-04 15:27:31.774054 I | raft: ccd4cb5684c5b1d4 became candidate at term 110
2018-12-04 15:27:31.774105 I | raft: ccd4cb5684c5b1d4 received MsgVoteResp from ccd4cb5684c5b1d4 at term 110
2018-12-04 15:27:31.774124 I | raft: ccd4cb5684c5b1d4 became leader at term 110
2018-12-04 15:27:31.774139 I | raft: raft.node: ccd4cb5684c5b1d4 elected leader ccd4cb5684c5b1d4 at term 110
2018-12-04 15:27:31.775580 I | etcdserver: published {Name:ip-X-X-X-X ClientURLs:[https://X.X.X.X:2379]} to cluster d36b3a36535103f8
2018-12-04 15:27:31.775647 I | embed: ready to serve client requests
2018-12-04 15:27:31.776818 I | embed: serving client requests on 10.0.4.57:2379
WARNING: 2018/12/04 15:27:32 Failed to dial X.X.X.X:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

Here are the logs from the crashed API server:

I1204 15:37:14.749434       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I1204 15:37:14.751323       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true NamespacedServiceBroker:true]
I1204 15:37:14.752882       1 hyperkube.go:192] Service Catalog version v3.11.0+58d854a-38;Upstream:v0.1.35 (built 2018-10-19T16:26:14Z)
W1204 15:37:15.953979       1 authentication.go:245] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: connection refused
[root@ip-10-0-4-57 ~]# docker logs e88bce8ff562
E1204 15:37:14.571957       1 helpers.go:134] Encountered config error json: unknown field "masterCount" in object *config.MasterConfig, raw JSON:
{"admissionConfig":{"pluginConfig":{"BuildDefaults":{"configuration":{"apiVersion":"v1","env":[],"kind":"BuildDefaultsConfig","resources":{"limits":{},"requests":{}}}},"BuildOverrides":{"configuration":{"apiVersion":"v1","kind":"BuildOverridesConfig"}},"openshift.io/ImagePolicy":{"configuration":{"apiVersion":"v1","executionRules":[{"matchImageAnnotations":[{"key":"images.openshift.io/deny-execution","value":"true"}],"name":"execution-denied","onResources":[{"resource":"pods"},{"resource":"builds"}],"reject":true,"skipOnResolutionFailure":true}],"kind":"ImagePolicyConfig"}}}},"aggregatorConfig":{"proxyClientInfo":{"certFile":"aggregator-front-proxy.crt","keyFile":"aggregator-front-proxy.key"}},"apiLevels":["v1"],"apiVersion":"v1","authConfig":{"requestHeader":{"clientCA":"front-proxy-ca.crt","clientCommonNames":["aggregator-front-proxy"],"extraHeaderPrefixes":["X-Remote-Extra-"],"groupHeaders":["X-Remote-Group"],"usernameHeaders":["X-Remote-User"]}},"controllerConfig":{"election":{"lockName":"openshift-master-controllers"},"serviceServingCert":{"signer":{"certFile":"service-signer.crt","keyFile":"service-signer.key"}}},"controllers":"*","corsAllowedOrigins":["(?i)//127\\.0\\.0\\.1(:|\\z)","(?i)//localhost(:|\\z)","(?i)//10\\.0\\.4\\.57(:|\\z)","(?i)//openshift\\.default\\.svc(:|\\z)","(?i)//os\\-int\\.tremolo\\.local(:|\\z)","(?i)//kubernetes\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//kubernetes(:|\\z)","(?i)//openshift\\.default(:|\\z)","(?i)//kubernetes\\.default(:|\\z)","(?i)//172\\.30\\.0\\.1(:|\\z)","(?i)//ip\\-10\\-0\\-4\\-57\\.ec2\\.internal(:|\\z)","(?i)//os\\.tremolo\\.io(:|\\z)","(?i)//openshift\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//kubernetes\\.default\\.svc(:|\\z)","(?i)//openshift(:|\\z)"],"dnsConfig":{"bindAddress":"0.0.0.0:8053","bindNetwork":"tcp4"},"etcdClientInfo":{"ca":"master.etcd-ca.crt","certFile":"master.etcd-client.crt","keyFile":"master.etcd-client.key","urls":["https://ip-10-0-4-57.ec2.internal:2379"]},"etcdStorageConfig":{"kubernetesStoragePrefix":"kubernetes.io","kubernetesStorageVersion":"v1","openShiftStoragePrefix":"openshift.io","openShiftStorageVersion":"v1"},"imageConfig":{"format":"docker.io/openshift/origin-${component}:${version}","latest":false},"imagePolicyConfig":{"internalRegistryHostname":"docker-registry.default.svc:5000"},"kind":"MasterConfig","kubeletClientInfo":{"ca":"ca-bundle.crt","certFile":"master.kubelet-client.crt","keyFile":"master.kubelet-client.key","port":10250},"kubernetesMasterConfig":{"apiServerArguments":{"cloud-config":["/etc/origin/cloudprovider/aws.conf"],"cloud-provider":["aws"],"runtime-config":[],"storage-backend":["etcd3"],"storage-media-type":["application/vnd.kubernetes.protobuf"]},"controllerArguments":{"cloud-config":["/etc/origin/cloudprovider/aws.conf"],"cloud-provider":["aws"],"cluster-signing-cert-file":["/etc/origin/master/ca.crt"],"cluster-signing-key-file":["/etc/origin/master/ca.key"],"pv-recycler-pod-template-filepath-hostpath":["/etc/origin/master/recycler_pod.yaml"],"pv-recycler-pod-template-filepath-nfs":["/etc/origin/master/recycler_pod.yaml"]},"masterCount":1,"masterIP":"10.0.4.57","podEvictionTimeout":null,"proxyClientInfo":{"certFile":"master.proxy-client.crt","keyFile":"master.proxy-client.key"},"schedulerArguments":null,"schedulerConfigFile":"/etc/origin/master/scheduler.json","servicesNodePortRange":"","servicesSubnet":"172.30.0.0/16","staticNodeNames":[]},"masterClients":{"externalKubernetesClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":400,"contentType":"application/vnd.kubernetes.protobuf","qps":200},"externalKubernetesKubeConfig":"","openshiftLoopbackClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":600,"contentType":"application/vnd.kubernetes.protobuf","qps":300},"openshiftLoopbackKubeConfig":"openshift-master.kubeconfig"},"masterPublicURL":"https://os.tremolo.io","networkConfig":{"clusterNetworks":[{"cidr":"10.128.0.0/14","hostSubnetLength":9}],"externalIPNetworkCIDRs":["0.0.0.0/0"],"networkPluginName":"redhat/openshift-ovs-subnet","serviceNetworkCIDR":"172.30.0.0/16"},"oauthConfig":{"assetPublicURL":"https://os.tremolo.io/console/","grantConfig":{"method":"auto"},"identityProviders":[{"challenge":false,"login":true,"mappingMethod":"claim","name":"unison","provider":{"apiVersion":"v1","claims":{"email":["email"],"id":["sub"],"name":["name"],"preferredUsername":["preferred_username"]},"clientID":"openshift","clientSecret":"8ncdvSgw6JUtRoI95P3c4ANcxHPwq7LYbz8s7NNQfnga0KNJIhApVVge0KSaxC4","kind":"OpenIDIdentityProvider","urls":{"authorize":"https://apps.tremolosecurity.com/auth/idp/OpenShiftIdP/auth","token":"https://apps.tremolosecurity.com/auth/idp/OpenShiftIdP/token"}}}],"masterCA":"ca-bundle.crt","masterPublicURL":"https://os.tremolo.io","masterURL":"https://os-int.tremolo.local","sessionConfig":{"sessionMaxAgeSeconds":3600,"sessionName":"ssn","sessionSecretsFile":"/etc/origin/master/session-secrets.yaml"},"tokenConfig":{"accessTokenMaxAgeSeconds":86400,"authorizeTokenMaxAgeSeconds":500}},"pauseControllers":false,"policyConfig":{"bootstrapPolicyFile":"/etc/origin/master/policy.json","openshiftInfrastructureNamespace":"openshift-infra","openshiftSharedResourcesNamespace":"openshift"},"projectConfig":{"defaultNodeSelector":"node-role.kubernetes.io/compute=true","projectRequestMessage":"","projectRequestTemplate":"","securityAllocator":{"mcsAllocatorRange":"s0:/2","mcsLabelsPerProject":5,"uidAllocatorRange":"1000000000-1999999999/10000"}},"routingConfig":{"subdomain":"router.default.svc.cluster.local"},"serviceAccountConfig":{"limitSecretReferences":false,"managedNames":["default","builder","deployer"],"masterCA":"ca-bundle.crt","privateKeyFile":"serviceaccounts.private.key","publicKeyFiles":["serviceaccounts.public.key"]},"servingInfo":{"bindAddress":"0.0.0.0:443","bindNetwork":"tcp4","certFile":"master.server.crt","clientCA":"ca.crt","keyFile":"master.server.key","maxRequestsInFlight":500,"requestTimeoutSeconds":3600},"volumeConfig":{"dynamicProvisioningEnabled":true}}
I1204 15:37:14.586394       1 plugins.go:84] Registered admission plugin "NamespaceLifecycle"
I1204 15:37:14.586436       1 plugins.go:84] Registered admission plugin "Initializers"
I1204 15:37:14.586448       1 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I1204 15:37:14.586504       1 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I1204 15:37:14.586516       1 plugins.go:84] Registered admission plugin "AlwaysAdmit"
I1204 15:37:14.586525       1 plugins.go:84] Registered admission plugin "AlwaysPullImages"
I1204 15:37:14.586535       1 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I1204 15:37:14.586546       1 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds"
I1204 15:37:14.586555       1 plugins.go:84] Registered admission plugin "AlwaysDeny"
I1204 15:37:14.586568       1 plugins.go:84] Registered admission plugin "EventRateLimit"
I1204 15:37:14.586579       1 plugins.go:84] Registered admission plugin "DenyEscalatingExec"
I1204 15:37:14.586587       1 plugins.go:84] Registered admission plugin "DenyExecOnPrivileged"
I1204 15:37:14.586597       1 plugins.go:84] Registered admission plugin "ExtendedResourceToleration"
I1204 15:37:14.586605       1 plugins.go:84] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I1204 15:37:14.586618       1 plugins.go:84] Registered admission plugin "ImagePolicyWebhook"
I1204 15:37:14.586629       1 plugins.go:84] Registered admission plugin "LimitRanger"
I1204 15:37:14.586640       1 plugins.go:84] Registered admission plugin "NamespaceAutoProvision"
I1204 15:37:14.586650       1 plugins.go:84] Registered admission plugin "NamespaceExists"
I1204 15:37:14.586660       1 plugins.go:84] Registered admission plugin "NodeRestriction"
I1204 15:37:14.586671       1 plugins.go:84] Registered admission plugin "PersistentVolumeLabel"
I1204 15:37:14.586681       1 plugins.go:84] Registered admission plugin "PodNodeSelector"
I1204 15:37:14.586691       1 plugins.go:84] Registered admission plugin "PodPreset"
I1204 15:37:14.586701       1 plugins.go:84] Registered admission plugin "PodTolerationRestriction"
I1204 15:37:14.586712       1 plugins.go:84] Registered admission plugin "ResourceQuota"
I1204 15:37:14.586722       1 plugins.go:84] Registered admission plugin "PodSecurityPolicy"
I1204 15:37:14.586730       1 plugins.go:84] Registered admission plugin "Priority"
I1204 15:37:14.586741       1 plugins.go:84] Registered admission plugin "SecurityContextDeny"
I1204 15:37:14.586752       1 plugins.go:84] Registered admission plugin "ServiceAccount"
I1204 15:37:14.586763       1 plugins.go:84] Registered admission plugin "DefaultStorageClass"
I1204 15:37:14.586773       1 plugins.go:84] Registered admission plugin "PersistentVolumeClaimResize"
I1204 15:37:14.586784       1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection"
F1204 15:37:44.610427       1 start_api.go:68] dial tcp 10.0.4.57:2379: connect: connection refused

Version

oc version
oc v3.11.0+62803d0-1
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://os-int.tremolo.local:443
openshift v3.11.0+06cfa24-67
kubernetes v1.11.0+d4cacc0

Steps To Reproduce

deploy okd 3.10
remove and disable ansible service broker
upgrade to 3.11

Current Result

the api server dies after a few minutes

Expected Result

etcd doesn't crash, api server doesn't crash

Additional Information

cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

The text was updated successfully, but these errors were encountered:

mlbiam · 2018-12-04T16:04:59Z

trying to recreate the etcd certs with playbooks/openshift-etcd/redeploy-certificates.yml which completed, but the same error. looking on the master the certs are still valid:

openssl x509 -in ./etcd.server.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7 (0x7)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=openshift-signer@1515411841
        Validity
            Not Before: Jan  8 11:44:03 2018 GMT
            Not After : Jan  8 11:44:04 2020 GMT
.
.
.

mlbiam · 2018-12-04T16:36:51Z

I disabled client auth in etcd by setting ETCD_CLIENT_CERT_AUTH=false, which got me past this error, however the api server is killing the etcd pod with the liveness probe failing:

Liveness probe failed: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""2 times in the last minute
--

mlbiam · 2018-12-04T17:14:38Z

I reenabled client auth, as I'm thinking the ssl error is a red herring. Instead I think the liveness probe isn't spawning correctly. Running the liveness probe from the command line works:

etcdctl --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt --endpoints https://X.X.X.X:2379 cluster-health
member ccd4cb5684c5b1d4 is healthy: got healthy result from https://X.X.X.X:2379
cluster is healthy

@vrutkovs asked to run docker exec directly, and I get the same error as the probe:

docker exec -ti cc1195d1292c bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

this seems to be JUST an issue with etcd though, as running with any other pod on the node works fine:

docker exec -ti 5364d8e20a2c bash
bash-4.2$ exit
exit

Looking in journalctl and it doesn't appear to have any new info:

Dec  4 17:13:45 ip-10-0-4-57 dockerd-current: time="2018-12-04T17:13:45.013787929Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:110: decoding init error from pipe caused \\\"read parent: connection reset by peer\\\"\"\n"
Dec  4 17:13:45 ip-10-0-4-57 dockerd-current: time="2018-12-04T17:13:45.014685767Z" level=error msg="Handler for POST /v1.26/exec/cadea80c910c2c768f30482c10d05e388f1c8c855f312d2181f880c1463f1644/resize returned error: rpc error: code = 2 desc = containerd: process not found for container"
Dec  4 17:13:45 ip-10-0-4-57 dockerd-current: time="2018-12-04T17:13:45.213612584Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:110: decoding init error from pipe caused \\\"read parent: connection reset by peer\\\"\"\n"
Dec  4 17:13:45 ip-10-0-4-57 origin-node: W1204 17:13:45.215045   17132 prober.go:103] No ref for container "docker://cc1195d1292c1c669e7e2659edaec00a8f4e06766d57c63e05eb6dc3b435672e" (master-etcd-ip-10-0-4-57.ec2.internal_kube-system(ef2a9d41f8b0e1fe5fc1ed96d4c3ce52):etcd)
Dec  4 17:13:55 ip-10-0-4-57 dockerd-current: time="2018-12-04T17:13:55.212705057Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:110: decoding init error from pipe caused \\\"read parent: connection reset by peer\\\"\"\n"
Dec  4 17:13:55 ip-10-0-4-57 origin-node: W1204 17:13:55.214343   17132 prober.go:103] No ref for container "docker://cc1195d1292c1c669e7e2659edaec00a8f4e06766d57c63e05eb6dc3b435672e" (master-etcd-ip-10-0-4-57.ec2.internal_kube-system(ef2a9d41f8b0e1fe5fc1ed96d4c3ce52):etcd)

my docker version is 1.13.1

lbrigman124 · 2018-12-04T17:25:35Z

We are seeing the same issue with the 3.10 and the liveness checks on the etcd pod. Our issues started on 11/29. Cannot exec into the etcd pod.

mlbiam · 2018-12-04T17:26:13Z

so it looks like the issue is related to CentOS 7.6 . talking to a customer with latest 3.11 on centos 7.5 is fine.

ghost · 2018-12-04T21:44:10Z

I'm having the same issues (#21575) not sure it's just CentOS 7.6 as I'm running RHEL 7.6, just tried deploying with RHEL 7.5 and I get the same issue.

Just like you I'm able to run the liveness probe from the command line and works fine

etcdctl --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt --endpoints https://XX.XXX.XX.XXX:2379 cluster-health
member e7e46ded4b8f5495 is healthy: got healthy result from https://XX.XXX.XX.XXX:2379 cluster is healthy

running docker exec give me the same error

docker exec -ti 90de6c2163c2 /bin/bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

alextremblay · 2018-12-04T22:28:42Z

We encountered a similar issue on one of our clusters after we updated docker to version 1.13.1-84.git07f3374.el7.centos
We resolved the problem be downgrading docker to version 1.13.1-75.git8633870.el7.centos

origin v3.10.0
centos 7.6

lbrigman124 · 2018-12-05T02:42:38Z

Seems to be a later release of docker at fault.
https://bugzilla.redhat.com/show_bug.cgi?id=1655975

ghost · 2018-12-05T03:33:19Z

Redhat's solution is to downgrading docker to version 1.13.1-75 (https://access.redhat.com/solutions/3734981) which fixed the issue with etcd crashing for me

rodrigovalin · 2018-12-06T16:45:20Z

Would you guys share RedHat's solution in this thread please? I don't have a Red Hat subscription.

Reamer · 2018-12-06T16:52:54Z

@rodrigovalin
RedHat's Solution is to downgrade docker to version 1.13.1-75, as @simongrzebieta mention

rodrigovalin · 2018-12-06T17:18:07Z

Solution:

In CentOS7 do: yum install docker-1.13.1-75.git8633870.el7.centos

This will downgrade the docker version to a working one.

zhachaoy · 2018-12-07T06:24:35Z

env: centos7.6(1810), openshift 3.11
etcd container exited every a few minutes
docker logs error:

2018-12-06 10:30:08.433154 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://192.168.10.127:2379
2018-12-06 10:30:08.433220 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/server.crt
2018-12-06 10:30:08.433225 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2018-12-06 10:30:08.433232 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/
2018-12-06 10:30:08.433235 I | pkg/flags: recognized and used environment variable ETCD_DEBUG=False
2018-12-06 10:30:08.433241 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=2500
2018-12-06 10:30:08.433247 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=500
2018-12-06 10:30:08.433275 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.10.127:2380
2018-12-06 10:30:08.433282 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=okd-master-1=https://192.168.10.127:2380,okd-master-2=https://192.168.10.128:2380
2018-12-06 10:30:08.433287 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2018-12-06 10:30:08.433291 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
2018-12-06 10:30:08.433299 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/server.key
2018-12-06 10:30:08.433304 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://192.168.10.127:2379
2018-12-06 10:30:08.433310 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://192.168.10.127:2380
2018-12-06 10:30:08.433318 I | pkg/flags: recognized and used environment variable ETCD_NAME=okd-master-1
2018-12-06 10:30:08.433323 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
2018-12-06 10:30:08.433327 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2018-12-06 10:30:08.433332 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
2018-12-06 10:30:08.433336 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-06 10:30:08.433342 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=4294967296
2018-12-06 10:30:08.433349 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-06 10:30:08.433374 I | etcdmain: etcd Version: 3.2.22
2018-12-06 10:30:08.433378 I | etcdmain: Git SHA: 1674e682f
2018-12-06 10:30:08.433381 I | etcdmain: Go Version: go1.8.7
2018-12-06 10:30:08.433384 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-06 10:30:08.433387 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2018-12-06 10:30:08.433434 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-12-06 10:30:08.433475 I | embed: peerTLS: cert = /etc/etcd/peer.crt, key = /etc/etcd/peer.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-06 10:30:08.433971 I | embed: listening for peers on https://192.168.10.127:2380
2018-12-06 10:30:08.434030 I | embed: listening for client requests on 192.168.10.127:2379
2018-12-06 10:30:08.436721 I | etcdserver: name = okd-master-1
2018-12-06 10:30:08.436727 I | etcdserver: data dir = /var/lib/etcd/
2018-12-06 10:30:08.436730 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-06 10:30:08.436733 I | etcdserver: heartbeat = 500ms
2018-12-06 10:30:08.436751 I | etcdserver: election = 2500ms
2018-12-06 10:30:08.436753 I | etcdserver: snapshot count = 100000
2018-12-06 10:30:08.436758 I | etcdserver: advertise client URLs = https://192.168.10.127:2379
2018-12-06 10:30:08.473611 I | etcdserver: restarting member 6e2b7e17eae1d8a3 in cluster bcf24f9d20a5bd30 at commit index 3802
2018-12-06 10:30:08.473844 I | raft: 6e2b7e17eae1d8a3 became follower at term 25
2018-12-06 10:30:08.473858 I | raft: newRaft 6e2b7e17eae1d8a3 [peers: [], term: 25, commit: 3802, applied: 0, lastindex: 3802, lastterm: 25]
2018-12-06 10:30:08.553577 I | mvcc: restore compact to 2312
2018-12-06 10:30:08.556937 W | auth: simple token is not cryptographically signed
2018-12-06 10:30:08.587664 I | etcdserver: starting server... [version: 3.2.22, cluster version: to_be_decided]
2018-12-06 10:30:08.588059 I | etcdserver/membership: added member 6e2b7e17eae1d8a3 [https://192.168.10.127:2380] to cluster bcf24f9d20a5bd30
2018-12-06 10:30:08.588120 I | etcdserver/membership: added member c4617694099e2b57 [https://192.168.10.128:2380] to cluster bcf24f9d20a5bd30
2018-12-06 10:30:08.588133 I | rafthttp: starting peer c4617694099e2b57...
2018-12-06 10:30:08.588142 I | rafthttp: started HTTP pipelining with peer c4617694099e2b57
2018-12-06 10:30:08.588196 I | embed: ClientTLS: cert = /etc/etcd/server.crt, key = /etc/etcd/server.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-06 10:30:08.588316 I | rafthttp: started streaming with peer c4617694099e2b57 (writer)
2018-12-06 10:30:08.588716 I | rafthttp: started streaming with peer c4617694099e2b57 (writer)
2018-12-06 10:30:08.588749 I | rafthttp: started peer c4617694099e2b57
2018-12-06 10:30:08.588779 I | rafthttp: started streaming with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:30:08.588805 I | rafthttp: started streaming with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:30:08.588874 I | rafthttp: added peer c4617694099e2b57
2018-12-06 10:30:08.589041 N | etcdserver/membership: set the initial cluster version to 3.2
2018-12-06 10:30:08.589135 I | rafthttp: peer c4617694099e2b57 became active
2018-12-06 10:30:08.589152 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream Message writer)
2018-12-06 10:30:08.589177 I | etcdserver/api: enabled capabilities for version 3.2
2018-12-06 10:30:08.590271 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 writer)
2018-12-06 10:30:08.597079 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:30:08.597546 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:30:08.738753 I | raft: raft.node: 6e2b7e17eae1d8a3 elected leader c4617694099e2b57 at term 25
2018-12-06 10:30:08.755641 I | embed: ready to serve client requests
2018-12-06 10:30:08.755740 I | etcdserver: published {Name:okd-master-1 ClientURLs:[https://192.168.10.127:2379]} to cluster bcf24f9d20a5bd30
2018-12-06 10:30:08.755872 I | embed: serving client requests on 192.168.10.127:2379
2018-12-06 10:30:21.849358 I | raft: 6e2b7e17eae1d8a3 [term 25] received MsgTimeoutNow from c4617694099e2b57 and starts an election to get leadership.
2018-12-06 10:30:21.849450 I | raft: 6e2b7e17eae1d8a3 became candidate at term 26
2018-12-06 10:30:21.849572 I | raft: 6e2b7e17eae1d8a3 received MsgVoteResp from 6e2b7e17eae1d8a3 at term 26
2018-12-06 10:30:21.849583 I | raft: 6e2b7e17eae1d8a3 [logterm: 25, index: 3855] sent MsgVote request to c4617694099e2b57 at term 26
2018-12-06 10:30:21.849589 I | raft: raft.node: 6e2b7e17eae1d8a3 lost leader c4617694099e2b57 at term 26
2018-12-06 10:30:21.874777 I | raft: 6e2b7e17eae1d8a3 received MsgVoteResp from c4617694099e2b57 at term 26
2018-12-06 10:30:21.874795 I | raft: 6e2b7e17eae1d8a3 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
2018-12-06 10:30:21.874807 I | raft: 6e2b7e17eae1d8a3 became leader at term 26
2018-12-06 10:30:21.874817 I | raft: raft.node: 6e2b7e17eae1d8a3 elected leader 6e2b7e17eae1d8a3 at term 26
2018-12-06 10:30:22.348742 W | rafthttp: lost the TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:30:22.348785 W | rafthttp: lost the TCP streaming connection with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:30:22.449539 E | rafthttp: failed to dial c4617694099e2b57 on stream Message (dial tcp 192.168.10.128:2380: getsockopt: connection refused)
2018-12-06 10:30:22.449552 I | rafthttp: peer c4617694099e2b57 became inactive
2018-12-06 10:30:23.474579 W | rafthttp: lost the TCP streaming connection with peer c4617694099e2b57 (stream Message writer)
2018-12-06 10:30:23.858891 I | rafthttp: peer c4617694099e2b57 became active
2018-12-06 10:30:23.858908 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:30:23.858963 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:30:23.866680 W | rafthttp: closed an existing TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 writer)
2018-12-06 10:30:23.866690 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 writer)
2018-12-06 10:30:23.868730 I | rafthttp: established a TCP streaming connection with peer c4617694099e2b57 (stream Message writer)
2018-12-06 10:31:16.394224 N | pkg/osutil: received terminated signal, shutting down...
WARNING: 2018/12/06 10:31:16 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.10.127:2379: getsockopt: connection refused"; Reconnecting to {192.168.10.127:2379 0 }
WARNING: 2018/12/06 10:31:17 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.10.127:2379: getsockopt: connection refused"; Reconnecting to {192.168.10.127:2379 0 }
WARNING: 2018/12/06 10:31:18 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.10.127:2379: getsockopt: connection refused"; Reconnecting to {192.168.10.127:2379 0 }
WARNING: 2018/12/06 10:31:21 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.10.127:2379: getsockopt: connection refused"; Reconnecting to {192.168.10.127:2379 0 }
WARNING: 2018/12/06 10:31:24 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.10.127:2379: getsockopt: connection refused"; Reconnecting to {192.168.10.127:2379 0 }
2018-12-06 10:31:26.394595 I | etcdserver: 6e2b7e17eae1d8a3 starts leadership transfer from 6e2b7e17eae1d8a3 to c4617694099e2b57
2018-12-06 10:31:26.394648 I | raft: 6e2b7e17eae1d8a3 [term 26] starts to transfer leadership to c4617694099e2b57
2018-12-06 10:31:26.394694 I | raft: 6e2b7e17eae1d8a3 sends MsgTimeoutNow to c4617694099e2b57 immediately as c4617694099e2b57 already has up-to-date log
2018-12-06 10:31:26.394904 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.394993 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395219 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395393 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395510 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395545 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395573 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395617 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395647 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395666 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395697 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
WARNING: 2018/12/06 10:31:26 grpc: addrConn.transportMonitor exits due to: context canceled
2018-12-06 10:31:26.395717 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395752 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395773 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395784 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395807 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395814 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395826 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395837 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395845 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395858 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395875 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395901 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395925 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395942 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395959 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.395992 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396007 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396016 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396026 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396049 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396063 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396115 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396139 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396169 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396183 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396191 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396207 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396265 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396297 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.396339 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-06 10:31:26.408813 I | raft: 6e2b7e17eae1d8a3 [term: 26] received a MsgVote message with higher term from c4617694099e2b57 [term: 27]
2018-12-06 10:31:26.408846 I | raft: 6e2b7e17eae1d8a3 became follower at term 27
2018-12-06 10:31:26.408853 I | raft: 6e2b7e17eae1d8a3 [logterm: 26, index: 4056, vote: 0] cast MsgVote for c4617694099e2b57 [logterm: 26, index: 4056] at term 27
2018-12-06 10:31:26.408859 I | raft: raft.node: 6e2b7e17eae1d8a3 lost leader 6e2b7e17eae1d8a3 at term 27
2018-12-06 10:31:26.420987 I | raft: raft.node: 6e2b7e17eae1d8a3 elected leader c4617694099e2b57 at term 27
2018-12-06 10:31:26.894897 I | etcdserver: 6e2b7e17eae1d8a3 finished leadership transfer from 6e2b7e17eae1d8a3 to c4617694099e2b57 (took 500.294632ms)
2018-12-06 10:31:26.894965 I | rafthttp: stopping peer c4617694099e2b57...
2018-12-06 10:31:26.895295 I | rafthttp: closed the TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 writer)
2018-12-06 10:31:26.895306 I | rafthttp: stopped streaming with peer c4617694099e2b57 (writer)
2018-12-06 10:31:26.895596 I | rafthttp: closed the TCP streaming connection with peer c4617694099e2b57 (stream Message writer)
2018-12-06 10:31:26.895606 I | rafthttp: stopped streaming with peer c4617694099e2b57 (writer)
2018-12-06 10:31:26.895675 I | rafthttp: stopped HTTP pipelining with peer c4617694099e2b57
2018-12-06 10:31:26.895779 W | rafthttp: lost the TCP streaming connection with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:31:26.895795 E | rafthttp: failed to read c4617694099e2b57 on stream MsgApp v2 (context canceled)
2018-12-06 10:31:26.895800 I | rafthttp: peer c4617694099e2b57 became inactive
2018-12-06 10:31:26.895808 I | rafthttp: stopped streaming with peer c4617694099e2b57 (stream MsgApp v2 reader)
2018-12-06 10:31:26.895858 W | rafthttp: lost the TCP streaming connection with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:31:26.895870 I | rafthttp: stopped streaming with peer c4617694099e2b57 (stream Message reader)
2018-12-06 10:31:26.895878 I | rafthttp: stopped peer c4617694099e2b57

vrutkovs · 2018-12-07T10:01:11Z

This is caused by docker bug - see https://bugzilla.redhat.com/show_bug.cgi?id=1655214

dharmit · 2018-12-15T04:48:31Z

We seem to be facing this issue even with the docker version mentioned in errata:

$ rpm -q docker
docker-1.13.1-88.git07f3374.el7.centos.x86_64

$ docker ps -a -f status=exited | grep etcd
3042d2bf6e82        ff5dd2137a4f      "/bin/sh -c '#!/bi..."   10 minutes ago      Exited (0) 2 minutes ago                         k8s_etcd_master-etcd-...

$ docker logs 3042d2bf6e82 2> etcd.log

$ cat etcd.log
2018-12-15 04:33:37.681961 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://172.22.0.69:2379
2018-12-15 04:33:37.682098 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/server.crt
2018-12-15 04:33:37.682112 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2018-12-15 04:33:37.682127 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/
2018-12-15 04:33:37.682152 I | pkg/flags: recognized and used environment variable ETCD_DEBUG=False
2018-12-15 04:33:37.682187 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=2500
2018-12-15 04:33:37.682245 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=500
2018-12-15 04:33:37.682282 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.22.0.69:2380
2018-12-15 04:33:37.682317 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=example.com=https://172.22.0.69:2380
2018-12-15 04:33:37.682334 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2018-12-15 04:33:37.682343 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
2018-12-15 04:33:37.682369 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/server.key
2018-12-15 04:33:37.682398 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://172.22.0.69:2379
2018-12-15 04:33:37.682427 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://172.22.0.69:2380
2018-12-15 04:33:37.682456 I | pkg/flags: recognized and used environment variable ETCD_NAME=example.com
2018-12-15 04:33:37.682487 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
2018-12-15 04:33:37.682511 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2018-12-15 04:33:37.682535 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
2018-12-15 04:33:37.682556 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-15 04:33:37.682590 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=4294967296
2018-12-15 04:33:37.682633 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-12-15 04:33:37.682738 I | etcdmain: etcd Version: 3.2.22
2018-12-15 04:33:37.682798 I | etcdmain: Git SHA: 1674e682f
2018-12-15 04:33:37.682809 I | etcdmain: Go Version: go1.8.7
2018-12-15 04:33:37.682823 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-15 04:33:37.682837 I | etcdmain: setting maximum number of CPUs to 16, total number of available CPUs is 16
2018-12-15 04:33:37.682934 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-12-15 04:33:37.683003 I | embed: peerTLS: cert = /etc/etcd/peer.crt, key = /etc/etcd/peer.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-15 04:33:37.684669 I | embed: listening for peers on https://172.22.0.69:2380
2018-12-15 04:33:37.684837 I | embed: listening for client requests on 172.22.0.69:2379
2018-12-15 04:33:37.696126 I | etcdserver: recovered store from snapshot at index 100001
2018-12-15 04:33:37.771467 I | mvcc: restore compact to 122212
2018-12-15 04:33:37.836858 I | etcdserver: name = example.com
2018-12-15 04:33:37.836923 I | etcdserver: data dir = /var/lib/etcd/
2018-12-15 04:33:37.836937 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-15 04:33:37.836944 I | etcdserver: heartbeat = 500ms
2018-12-15 04:33:37.836950 I | etcdserver: election = 2500ms
2018-12-15 04:33:37.836957 I | etcdserver: snapshot count = 100000
2018-12-15 04:33:37.836984 I | etcdserver: advertise client URLs = https://172.22.0.69:2379
2018-12-15 04:33:39.167607 I | etcdserver: restarting member 9183452eeea616c8 in cluster e6e44ef9ce2e9b6f at commit index 144682
2018-12-15 04:33:39.170592 I | raft: 9183452eeea616c8 became follower at term 19
2018-12-15 04:33:39.170651 I | raft: newRaft 9183452eeea616c8 [peers: [9183452eeea616c8], term: 19, commit: 144682, applied: 100001, lastindex: 144682, lastterm: 19]
2018-12-15 04:33:39.170969 I | etcdserver/api: enabled capabilities for version 3.2
2018-12-15 04:33:39.171022 I | etcdserver/membership: added member 9183452eeea616c8 [https://172.22.0.69:2380] to cluster e6e44ef9ce2e9b6f from store
2018-12-15 04:33:39.171041 I | etcdserver/membership: set the cluster version to 3.2 from store
2018-12-15 04:33:39.357462 I | mvcc: restore compact to 122212
2018-12-15 04:33:39.401136 W | auth: simple token is not cryptographically signed
2018-12-15 04:33:39.574613 I | etcdserver: starting server... [version: 3.2.22, cluster version: 3.2]
2018-12-15 04:33:39.575412 I | etcdserver: 9183452eeea616c8 as single-node; fast-forwarding 4 ticks (election ticks 5)
2018-12-15 04:33:39.575679 I | embed: ClientTLS: cert = /etc/etcd/server.crt, key = /etc/etcd/server.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-12-15 04:33:40.671617 I | raft: 9183452eeea616c8 is starting a new election at term 19
2018-12-15 04:33:40.671687 I | raft: 9183452eeea616c8 became candidate at term 20
2018-12-15 04:33:40.671771 I | raft: 9183452eeea616c8 received MsgVoteResp from 9183452eeea616c8 at term 20
2018-12-15 04:33:40.671798 I | raft: 9183452eeea616c8 became leader at term 20
2018-12-15 04:33:40.671817 I | raft: raft.node: 9183452eeea616c8 elected leader 9183452eeea616c8 at term 20
2018-12-15 04:33:40.672307 I | etcdserver: published {Name:example.com ClientURLs:[https://172.22.0.69:2379]} to cluster e6e44ef9ce2e9b6f
2018-12-15 04:33:40.672424 I | embed: ready to serve client requests
2018-12-15 04:33:40.672868 I | embed: serving client requests on 172.22.0.69:2379
2018-12-15 04:34:09.079912 W | etcdserver: apply entries took too long [147.724949ms for 1 entries]
2018-12-15 04:34:09.079949 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:10.420081 W | etcdserver: apply entries took too long [266.889229ms for 1 entries]
2018-12-15 04:34:10.420116 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:10.712956 W | etcdserver: apply entries took too long [146.532473ms for 1 entries]
2018-12-15 04:34:10.713004 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:14.390582 W | etcdserver: apply entries took too long [180.439125ms for 1 entries]
2018-12-15 04:34:14.390625 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:15.495040 W | etcdserver: apply entries took too long [329.845995ms for 1 entries]
2018-12-15 04:34:15.495082 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:16.740341 W | etcdserver: apply entries took too long [163.457997ms for 1 entries]
2018-12-15 04:34:16.740375 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:19.912822 W | etcdserver: apply entries took too long [108.854698ms for 1 entries]
2018-12-15 04:34:19.912866 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:20.397934 W | etcdserver: apply entries took too long [108.615192ms for 1 entries]
2018-12-15 04:34:20.397968 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:38.461049 W | etcdserver: apply entries took too long [3.371763838s for 1 entries]
2018-12-15 04:34:38.461081 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:45.935182 W | wal: sync duration of 1.280364025s, expected less than 1s
2018-12-15 04:34:48.267088 W | wal: sync duration of 1.138972376s, expected less than 1s
2018-12-15 04:34:48.789631 W | etcdserver: apply entries took too long [239.488803ms for 1 entries]
2018-12-15 04:34:48.789657 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:50.782953 W | etcdserver: apply entries took too long [110.122946ms for 1 entries]
2018-12-15 04:34:50.782997 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:51.033497 W | etcdserver: apply entries took too long [127.534381ms for 1 entries]
2018-12-15 04:34:51.033531 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:34:55.871999 W | etcdserver: failed to revoke 16c867b0242968b8 ("lease not found")
2018-12-15 04:34:55.872065 W | etcdserver: failed to revoke 16c867b0242968b8 ("lease not found")
2018-12-15 04:34:55.872642 W | etcdserver: failed to revoke 16c867b0242968b8 ("lease not found")
2018-12-15 04:35:22.270278 W | etcdserver: apply entries took too long [113.085996ms for 1 entries]
2018-12-15 04:35:22.270329 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:22.490805 W | etcdserver: apply entries took too long [117.023281ms for 1 entries]
2018-12-15 04:35:22.490849 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:23.005640 W | etcdserver: apply entries took too long [175.383529ms for 1 entries]
2018-12-15 04:35:23.005686 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:34.078661 W | wal: sync duration of 1.10625727s, expected less than 1s
2018-12-15 04:35:35.766285 W | etcdserver: apply entries took too long [430.55193ms for 1 entries]
2018-12-15 04:35:35.766343 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:36.735074 W | etcdserver: apply entries took too long [124.059531ms for 1 entries]
2018-12-15 04:35:36.735113 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:41.104198 W | etcdserver: apply entries took too long [161.952077ms for 1 entries]
2018-12-15 04:35:41.104227 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:49.175358 W | wal: sync duration of 1.8133252s, expected less than 1s
2018-12-15 04:35:51.521223 W | etcdserver: apply entries took too long [377.591351ms for 1 entries]
2018-12-15 04:35:51.521254 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:51.800223 W | etcdserver: apply entries took too long [173.260619ms for 1 entries]
2018-12-15 04:35:51.800299 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:35:56.617502 W | etcdserver: apply entries took too long [205.055586ms for 1 entries]
2018-12-15 04:35:56.617536 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:00.875166 W | wal: sync duration of 3.302501749s, expected less than 1s
2018-12-15 04:36:01.172517 W | etcdserver: apply entries took too long [297.005989ms for 2 entries]
2018-12-15 04:36:01.172555 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:01.597168 W | etcdserver: apply entries took too long [144.421291ms for 1 entries]
2018-12-15 04:36:01.597200 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:02.112186 W | etcdserver: apply entries took too long [150.762837ms for 1 entries]
2018-12-15 04:36:02.112239 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:04.793417 W | etcdserver: apply entries took too long [837.336053ms for 1 entries]
2018-12-15 04:36:04.793454 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:06.749835 W | wal: sync duration of 1.218318469s, expected less than 1s
2018-12-15 04:36:07.737132 W | etcdserver: failed to revoke 16c867b024296eb2 ("lease not found")
2018-12-15 04:36:10.318877 W | wal: sync duration of 1.086361054s, expected less than 1s
2018-12-15 04:36:12.962948 W | wal: sync duration of 1.463913925s, expected less than 1s
2018-12-15 04:36:16.182386 W | etcdserver: apply entries took too long [447.360451ms for 1 entries]
2018-12-15 04:36:16.182420 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:16.440403 W | etcdserver: apply entries took too long [157.682392ms for 1 entries]
2018-12-15 04:36:16.440435 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:17.590236 W | etcdserver: apply entries took too long [229.643293ms for 1 entries]
2018-12-15 04:36:17.590301 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:18.959592 W | wal: sync duration of 1.599104501s, expected less than 1s
2018-12-15 04:36:26.216192 W | wal: sync duration of 1.148563425s, expected less than 1s
2018-12-15 04:36:26.992127 W | etcdserver: apply entries took too long [164.770523ms for 1 entries]
2018-12-15 04:36:26.992166 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:28.823639 W | etcdserver: apply entries took too long [164.004523ms for 1 entries]
2018-12-15 04:36:28.823708 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:34.385464 W | wal: sync duration of 1.05855292s, expected less than 1s
2018-12-15 04:36:34.874010 W | etcdserver: apply entries took too long [116.094711ms for 1 entries]
2018-12-15 04:36:34.874051 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:45.440007 W | wal: sync duration of 1.162764613s, expected less than 1s
2018-12-15 04:36:48.438068 W | wal: sync duration of 1.587935042s, expected less than 1s
2018-12-15 04:36:50.557388 W | etcdserver: failed to revoke 16c867b02429719d ("lease not found")
2018-12-15 04:36:56.726950 W | wal: sync duration of 5.688342194s, expected less than 1s
2018-12-15 04:36:57.964213 W | etcdserver: apply entries took too long [338.987717ms for 2 entries]
2018-12-15 04:36:57.964247 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:36:58.359300 W | etcdserver: apply entries took too long [153.812991ms for 1 entries]
2018-12-15 04:36:58.359340 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:01.927143 W | etcdserver: apply entries took too long [2.596348519s for 5 entries]
2018-12-15 04:37:01.927180 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:02.108330 W | wal: sync duration of 2.777625582s, expected less than 1s
2018-12-15 04:37:02.447163 W | etcdserver: failed to revoke 16c867b024297395 ("lease not found")
2018-12-15 04:37:02.447194 W | etcdserver: failed to revoke 16c867b024297395 ("lease not found")
2018-12-15 04:37:02.529161 W | etcdserver: failed to revoke 16c867b024297395 ("lease not found")
2018-12-15 04:37:04.872494 W | wal: sync duration of 1.261677249s, expected less than 1s
2018-12-15 04:37:05.181482 W | etcdserver: apply entries took too long [308.708267ms for 3 entries]
2018-12-15 04:37:05.181519 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:07.782673 W | etcdserver: apply entries took too long [1.033221053s for 1 entries]
2018-12-15 04:37:07.782923 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:07.843167 W | wal: sync duration of 1.094166065s, expected less than 1s
2018-12-15 04:37:08.754178 W | etcdserver: apply entries took too long [210.874983ms for 1 entries]
2018-12-15 04:37:08.754584 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:10.929106 W | etcdserver: apply entries took too long [254.841897ms for 2 entries]
2018-12-15 04:37:10.929146 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:11.382995 W | etcdserver: apply entries took too long [332.356104ms for 1 entries]
2018-12-15 04:37:11.383046 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:14.706134 W | etcdserver: apply entries took too long [277.674898ms for 1 entries]
2018-12-15 04:37:14.706431 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:16.438071 W | wal: sync duration of 1.439587455s, expected less than 1s
2018-12-15 04:37:16.946964 W | etcdserver: apply entries took too long [323.775929ms for 2 entries]
2018-12-15 04:37:16.947006 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:19.172020 W | wal: sync duration of 1.456913903s, expected less than 1s
2018-12-15 04:37:19.223663 W | etcdserver: apply entries took too long [1.540646174s for 1 entries]
2018-12-15 04:37:19.223702 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:19.504626 W | etcdserver: apply entries took too long [176.315455ms for 1 entries]
2018-12-15 04:37:19.504661 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:21.382431 W | etcdserver: apply entries took too long [943.370067ms for 1 entries]
2018-12-15 04:37:21.382488 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:21.618186 W | etcdserver: apply entries took too long [103.585644ms for 1 entries]
2018-12-15 04:37:21.618241 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:22.377250 W | etcdserver: apply entries took too long [264.462444ms for 1 entries]
2018-12-15 04:37:22.377316 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:26.285685 W | etcdserver: apply entries took too long [1.108576474s for 1 entries]
2018-12-15 04:37:26.285737 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:26.805587 W | etcdserver: apply entries took too long [418.809911ms for 4 entries]
2018-12-15 04:37:26.805637 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:27.391245 W | etcdserver: apply entries took too long [103.040098ms for 1 entries]
2018-12-15 04:37:27.391305 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:33.342477 W | etcdserver: apply entries took too long [4.080687591s for 11 entries]
2018-12-15 04:37:33.342518 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:33.398062 W | wal: sync duration of 4.136231954s, expected less than 1s
2018-12-15 04:37:38.674635 W | etcdserver: apply entries took too long [193.443499ms for 1 entries]
2018-12-15 04:37:38.674684 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:39.822440 W | etcdserver: apply entries took too long [151.532537ms for 1 entries]
2018-12-15 04:37:39.822526 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:40.960043 W | etcdserver: apply entries took too long [904.461802ms for 1 entries]
2018-12-15 04:37:40.960116 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:41.297239 W | etcdserver: apply entries took too long [126.225998ms for 1 entries]
2018-12-15 04:37:41.297308 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:45.315441 W | wal: sync duration of 1.719181385s, expected less than 1s
2018-12-15 04:37:45.366458 W | etcdserver: apply entries took too long [1.770190934s for 11 entries]
2018-12-15 04:37:45.366502 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:47.837194 W | wal: sync duration of 1.359101724s, expected less than 1s
2018-12-15 04:37:48.460075 W | etcdserver: apply entries took too long [126.44647ms for 1 entries]
2018-12-15 04:37:48.460106 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:48.886204 W | etcdserver: apply entries took too long [246.647078ms for 1 entries]
2018-12-15 04:37:48.886238 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:50.757794 W | wal: sync duration of 1.529638954s, expected less than 1s
2018-12-15 04:37:50.758503 W | etcdserver: failed to revoke 16c867b0242975bf ("lease not found")
2018-12-15 04:37:50.758959 W | etcdserver: failed to revoke 16c867b0242975bf ("lease not found")
2018-12-15 04:37:51.044508 W | etcdserver: apply entries took too long [154.367383ms for 1 entries]
2018-12-15 04:37:51.044539 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:52.133282 W | etcdserver: apply entries took too long [171.349133ms for 1 entries]
2018-12-15 04:37:52.133333 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:53.172998 W | wal: sync duration of 1.109320555s, expected less than 1s
2018-12-15 04:37:57.988224 W | etcdserver: apply entries took too long [311.568584ms for 1 entries]
2018-12-15 04:37:57.988258 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:58.389480 W | etcdserver: apply entries took too long [133.870002ms for 1 entries]
2018-12-15 04:37:58.389513 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:59.514340 W | etcdserver: apply entries took too long [979.340028ms for 1 entries]
2018-12-15 04:37:59.514376 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:37:59.897285 W | etcdserver: apply entries took too long [191.775266ms for 1 entries]
2018-12-15 04:37:59.897828 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:01.955201 W | wal: sync duration of 2.00747001s, expected less than 1s
2018-12-15 04:38:03.147567 W | etcdserver: apply entries took too long [1.192035357s for 1 entries]
2018-12-15 04:38:03.147596 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:04.085307 W | etcdserver: apply entries took too long [133.396489ms for 1 entries]
2018-12-15 04:38:04.085352 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:05.559077 W | etcdserver: apply entries took too long [144.224004ms for 1 entries]
2018-12-15 04:38:05.559123 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:06.612484 W | etcdserver: apply entries took too long [280.527443ms for 1 entries]
2018-12-15 04:38:06.612525 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:08.132681 W | wal: sync duration of 1.26118986s, expected less than 1s
2018-12-15 04:38:08.240455 W | etcdserver: apply entries took too long [107.459521ms for 1 entries]
2018-12-15 04:38:08.240486 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:08.539824 W | etcdserver: apply entries took too long [188.139742ms for 1 entries]
2018-12-15 04:38:08.539874 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:08.951451 W | etcdserver: apply entries took too long [134.801659ms for 1 entries]
2018-12-15 04:38:08.951492 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:12.465852 W | etcdserver: apply entries took too long [122.759169ms for 1 entries]
2018-12-15 04:38:12.465887 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:13.622961 W | wal: sync duration of 1.039700892s, expected less than 1s
2018-12-15 04:38:13.646353 W | etcdserver: apply entries took too long [1.062997072s for 1 entries]
2018-12-15 04:38:13.646386 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:13.998218 W | etcdserver: apply entries took too long [238.386524ms for 1 entries]
2018-12-15 04:38:13.998278 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:15.538179 W | etcdserver: apply entries took too long [539.334857ms for 1 entries]
2018-12-15 04:38:15.538487 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:15.538563 W | etcdserver: failed to revoke 16c867b02429772b ("lease not found")
2018-12-15 04:38:19.721527 W | wal: sync duration of 2.77056893s, expected less than 1s
2018-12-15 04:38:20.790956 W | etcdserver: apply entries took too long [222.600613ms for 1 entries]
2018-12-15 04:38:20.790988 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:24.610148 W | wal: sync duration of 1.260978222s, expected less than 1s
2018-12-15 04:38:27.180886 W | etcdserver: apply entries took too long [1.186734484s for 10 entries]
2018-12-15 04:38:27.180920 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:27.192007 W | wal: sync duration of 1.197918018s, expected less than 1s
2018-12-15 04:38:29.501238 W | etcdserver: apply entries took too long [496.633556ms for 1 entries]
2018-12-15 04:38:29.501504 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:32.365961 W | etcdserver: apply entries took too long [2.679132591s for 1 entries]
2018-12-15 04:38:32.365997 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:33.311912 W | etcdserver: apply entries took too long [123.678255ms for 1 entries]
2018-12-15 04:38:33.311944 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:33.609558 W | etcdserver: apply entries took too long [195.971736ms for 1 entries]
2018-12-15 04:38:33.609607 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:34.926049 W | etcdserver: apply entries took too long [995.263237ms for 2 entries]
2018-12-15 04:38:34.926095 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:34.958338 W | wal: sync duration of 1.027649179s, expected less than 1s
2018-12-15 04:38:38.064410 W | etcdserver: apply entries took too long [185.118826ms for 1 entries]
2018-12-15 04:38:38.064459 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:38.789001 W | etcdserver: apply entries took too long [372.621992ms for 1 entries]
2018-12-15 04:38:38.789155 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:40.127397 W | wal: sync duration of 1.068574505s, expected less than 1s
2018-12-15 04:38:40.278264 W | etcdserver: apply entries took too long [150.604775ms for 1 entries]
2018-12-15 04:38:40.278336 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:40.743496 W | etcdserver: apply entries took too long [321.343745ms for 1 entries]
2018-12-15 04:38:40.743541 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:44.501399 W | wal: sync duration of 3.201018599s, expected less than 1s
2018-12-15 04:38:44.713381 W | etcdserver: apply entries took too long [211.560481ms for 2 entries]
2018-12-15 04:38:44.713432 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:45.023843 W | etcdserver: apply entries took too long [209.871032ms for 1 entries]
2018-12-15 04:38:45.023881 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:45.647181 W | etcdserver: apply entries took too long [174.096318ms for 1 entries]
2018-12-15 04:38:45.647225 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:47.769595 W | etcdserver: apply entries took too long [1.693709953s for 2 entries]
2018-12-15 04:38:47.769634 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:47.786889 W | wal: sync duration of 1.711072785s, expected less than 1s
2018-12-15 04:38:50.046983 W | wal: sync duration of 1.191499065s, expected less than 1s
2018-12-15 04:38:50.385066 W | etcdserver: apply entries took too long [136.33045ms for 1 entries]
2018-12-15 04:38:50.385099 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:50.620432 W | etcdserver: apply entries took too long [101.228443ms for 1 entries]
2018-12-15 04:38:50.620464 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:51.172289 W | etcdserver: apply entries took too long [167.363907ms for 1 entries]
2018-12-15 04:38:51.172364 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:52.519206 W | etcdserver: apply entries took too long [1.244868164s for 1 entries]
2018-12-15 04:38:52.519244 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:53.061092 W | etcdserver: apply entries took too long [133.588979ms for 1 entries]
2018-12-15 04:38:53.061166 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:54.618900 W | wal: sync duration of 1.065901654s, expected less than 1s
2018-12-15 04:38:54.660645 W | etcdserver: apply entries took too long [1.107451758s for 2 entries]
2018-12-15 04:38:54.660673 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:38:54.660709 W | etcdserver: failed to revoke 16c867b024297923 ("lease not found")
2018-12-15 04:38:54.660926 W | etcdserver: failed to revoke 16c867b024297923 ("lease not found")
2018-12-15 04:38:56.140434 W | wal: sync duration of 1.436991437s, expected less than 1s
2018-12-15 04:38:59.416943 W | wal: sync duration of 1.792751606s, expected less than 1s
2018-12-15 04:39:00.202546 W | etcdserver: apply entries took too long [289.162066ms for 1 entries]
2018-12-15 04:39:00.202586 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:02.004820 W | wal: sync duration of 1.25740636s, expected less than 1s
2018-12-15 04:39:02.457173 W | etcdserver: apply entries took too long [207.893149ms for 1 entries]
2018-12-15 04:39:02.457228 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:03.188955 W | etcdserver: apply entries took too long [192.137584ms for 1 entries]
2018-12-15 04:39:03.189004 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:04.693353 W | wal: sync duration of 1.070886438s, expected less than 1s
2018-12-15 04:39:08.677843 W | wal: sync duration of 2.761959292s, expected less than 1s
2018-12-15 04:39:12.036079 W | wal: sync duration of 1.693793652s, expected less than 1s
2018-12-15 04:39:14.079199 W | etcdserver: apply entries took too long [427.421754ms for 4 entries]
2018-12-15 04:39:14.079260 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:17.004400 W | wal: sync duration of 2.126993298s, expected less than 1s
2018-12-15 04:39:17.075270 W | etcdserver: apply entries took too long [2.197863805s for 2 entries]
2018-12-15 04:39:17.075325 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:17.930417 W | etcdserver: apply entries took too long [107.903336ms for 1 entries]
2018-12-15 04:39:17.930468 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:19.722570 W | wal: sync duration of 1.399253714s, expected less than 1s
2018-12-15 04:39:20.603256 W | etcdserver: apply entries took too long [174.928026ms for 1 entries]
2018-12-15 04:39:20.603549 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:22.406847 W | wal: sync duration of 1.470084296s, expected less than 1s
2018-12-15 04:39:22.897615 W | etcdserver: apply entries took too long [323.653234ms for 1 entries]
2018-12-15 04:39:22.897654 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:23.646212 W | etcdserver: apply entries took too long [242.969429ms for 2 entries]
2018-12-15 04:39:23.646255 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:26.701514 W | wal: sync duration of 2.981865043s, expected less than 1s
2018-12-15 04:39:28.026962 W | etcdserver: apply entries took too long [228.260792ms for 1 entries]
2018-12-15 04:39:28.027012 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:30.291782 W | wal: sync duration of 1.20631194s, expected less than 1s
2018-12-15 04:39:30.359899 W | etcdserver: apply entries took too long [1.274278871s for 1 entries]
2018-12-15 04:39:30.360051 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:30.360353 W | etcdserver: failed to revoke 16c867b024297ab5 ("lease not found")
2018-12-15 04:39:30.363649 W | etcdserver: failed to revoke 16c867b024297ab5 ("lease not found")
2018-12-15 04:39:30.704827 W | etcdserver: apply entries took too long [242.124983ms for 1 entries]
2018-12-15 04:39:30.704874 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:33.046856 W | wal: sync duration of 1.550213433s, expected less than 1s
2018-12-15 04:39:33.387250 W | etcdserver: apply entries took too long [141.265026ms for 1 entries]
2018-12-15 04:39:33.387293 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:36.453684 W | etcdserver: apply entries took too long [1.920298767s for 2 entries]
2018-12-15 04:39:36.453727 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:36.462300 W | wal: sync duration of 1.911447391s, expected less than 1s
2018-12-15 04:39:36.762734 W | etcdserver: apply entries took too long [161.321475ms for 1 entries]
2018-12-15 04:39:36.762798 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:40.255486 W | wal: sync duration of 2.969903353s, expected less than 1s
2018-12-15 04:39:41.347441 W | etcdserver: apply entries took too long [123.582677ms for 1 entries]
2018-12-15 04:39:41.347494 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:42.808519 W | wal: sync duration of 1.178710863s, expected less than 1s
2018-12-15 04:39:43.392691 W | etcdserver: apply entries took too long [120.117082ms for 1 entries]
2018-12-15 04:39:43.392724 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:43.964971 W | etcdserver: apply entries took too long [199.031613ms for 1 entries]
2018-12-15 04:39:43.965008 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:45.488823 W | wal: sync duration of 1.193343261s, expected less than 1s
2018-12-15 04:39:45.516088 W | etcdserver: apply entries took too long [1.220506245s for 2 entries]
2018-12-15 04:39:45.516130 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:47.881285 W | wal: sync duration of 1.090625813s, expected less than 1s
2018-12-15 04:39:48.412114 W | etcdserver: apply entries took too long [137.567743ms for 1 entries]
2018-12-15 04:39:48.412161 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:50.240882 W | etcdserver: apply entries took too long [237.589151ms for 1 entries]
2018-12-15 04:39:50.240939 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:51.839712 W | wal: sync duration of 1.573190956s, expected less than 1s
2018-12-15 04:39:53.482714 W | etcdserver: apply entries took too long [106.045634ms for 1 entries]
2018-12-15 04:39:53.482782 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:55.546037 W | etcdserver: apply entries took too long [138.964972ms for 1 entries]
2018-12-15 04:39:55.546079 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:56.229556 W | etcdserver: apply entries took too long [138.106953ms for 1 entries]
2018-12-15 04:39:56.229591 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:39:59.164906 W | wal: sync duration of 2.609477764s, expected less than 1s
2018-12-15 04:39:59.927491 W | etcdserver: apply entries took too long [3.371895096s for 1 entries]
2018-12-15 04:39:59.927517 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:00.905535 W | etcdserver: apply entries took too long [143.429925ms for 1 entries]
2018-12-15 04:40:00.905572 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:03.029569 W | etcdserver: apply entries took too long [1.162787758s for 1 entries]
2018-12-15 04:40:03.029613 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:03.107079 W | wal: sync duration of 1.240602626s, expected less than 1s
2018-12-15 04:40:04.133990 W | etcdserver: apply entries took too long [142.86047ms for 1 entries]
2018-12-15 04:40:04.134036 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:05.661033 W | wal: sync duration of 1.515223531s, expected less than 1s
2018-12-15 04:40:06.083086 W | etcdserver: apply entries took too long [228.715934ms for 2 entries]
2018-12-15 04:40:06.083119 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:07.848763 W | etcdserver: apply entries took too long [1.054470372s for 1 entries]
2018-12-15 04:40:07.848796 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:08.244994 W | etcdserver: apply entries took too long [107.227105ms for 1 entries]
2018-12-15 04:40:08.245038 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:11.070205 W | wal: sync duration of 1.351605146s, expected less than 1s
2018-12-15 04:40:11.169765 W | etcdserver: apply entries took too long [1.451094464s for 1 entries]
2018-12-15 04:40:11.172840 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:14.358186 W | etcdserver: apply entries took too long [260.726698ms for 2 entries]
2018-12-15 04:40:14.358230 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:14.599605 W | etcdserver: apply entries took too long [140.861938ms for 1 entries]
2018-12-15 04:40:14.599637 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:16.009050 W | etcdserver: apply entries took too long [541.616419ms for 4 entries]
2018-12-15 04:40:16.009175 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:16.519988 W | etcdserver: apply entries took too long [371.161048ms for 1 entries]
2018-12-15 04:40:16.520019 W | etcdserver: avoid queries with large range/delete range!
2018-12-15 04:40:18.112503 N | pkg/osutil: received terminated signal, shutting down...
WARNING: 2018/12/15 04:40:18 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 172.22.0.69:44560->172.22.0.69:2379: read: connection reset by peer"; Reconnecting to {172.22.0.69:2379 0  <nil>}
2018-12-15 04:40:18.316472 W | wal: sync duration of 1.765336193s, expected less than 1s
WARNING: 2018/12/15 04:40:19 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 172.22.0.69:2379: getsockopt: connection refused"; Reconnecting to {172.22.0.69:2379 0  <nil>}
WARNING: 2018/12/15 04:40:20 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 172.22.0.69:2379: getsockopt: connection refused"; Reconnecting to {172.22.0.69:2379 0  <nil>}
WARNING: 2018/12/15 04:40:23 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 172.22.0.69:2379: getsockopt: connection refused"; Reconnecting to {172.22.0.69:2379 0  <nil>}
2018-12-15 04:40:28.113007 I | etcdserver: skipped leadership transfer for single member cluster
2018-12-15 04:40:28.113159 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.113364 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.113432 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.113473 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.113506 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.113549 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
WARNING: 2018/12/15 04:40:28 grpc: addrConn.transportMonitor exits due to: context canceled
2018-12-15 04:40:28.114149 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114350 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114433 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114460 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114496 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114515 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114540 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114591 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114607 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114651 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114762 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114835 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114857 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114942 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.114998 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115057 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115185 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115247 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115274 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115439 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115474 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115493 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115532 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115662 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115721 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115762 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115807 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.115953 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.116061 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
2018-12-15 04:40:28.116242 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")

mlbiam changed the title ~~etcd crashing on upgraded 3.11 instance~~ etcd crashing on upgraded 3.11 instance due to failing liveness probe Dec 4, 2018

vrutkovs closed this as completed Dec 7, 2018

stewartshea mentioned this issue Dec 9, 2018

OpenShift 3.11 CentOS7 becomes unresponsive etcd problem etcd-io/etcd#10310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd crashing on upgraded 3.11 instance due to failing liveness probe #21609

etcd crashing on upgraded 3.11 instance due to failing liveness probe #21609

mlbiam commented Dec 4, 2018

mlbiam commented Dec 4, 2018

mlbiam commented Dec 4, 2018

mlbiam commented Dec 4, 2018

lbrigman124 commented Dec 4, 2018

mlbiam commented Dec 4, 2018

ghost commented Dec 4, 2018 •

edited by ghost

Loading

alextremblay commented Dec 4, 2018

lbrigman124 commented Dec 5, 2018

ghost commented Dec 5, 2018

rodrigovalin commented Dec 6, 2018

Reamer commented Dec 6, 2018

rodrigovalin commented Dec 6, 2018 •

edited

Loading

zhachaoy commented Dec 7, 2018 •

edited

Loading

vrutkovs commented Dec 7, 2018

dharmit commented Dec 15, 2018

etcd crashing on upgraded 3.11 instance due to failing liveness probe #21609

etcd crashing on upgraded 3.11 instance due to failing liveness probe #21609

Comments

mlbiam commented Dec 4, 2018

Version

Steps To Reproduce

Current Result

Expected Result

Additional Information

mlbiam commented Dec 4, 2018

mlbiam commented Dec 4, 2018

mlbiam commented Dec 4, 2018

lbrigman124 commented Dec 4, 2018

mlbiam commented Dec 4, 2018

ghost commented Dec 4, 2018 • edited by ghost Loading

alextremblay commented Dec 4, 2018

lbrigman124 commented Dec 5, 2018

ghost commented Dec 5, 2018

rodrigovalin commented Dec 6, 2018

Reamer commented Dec 6, 2018

rodrigovalin commented Dec 6, 2018 • edited Loading

zhachaoy commented Dec 7, 2018 • edited Loading

vrutkovs commented Dec 7, 2018

dharmit commented Dec 15, 2018

ghost commented Dec 4, 2018 •

edited by ghost

Loading

rodrigovalin commented Dec 6, 2018 •

edited

Loading

zhachaoy commented Dec 7, 2018 •

edited

Loading