Investigate etcd unavailability leading to 300s cluster up timeout

[bprashanth]

#20931 (comment)
Apiserver logs:

E0306 01:08:48.052514       6 cacher.go:217] unexpected ListAndWatch error: pkg/storage/cacher.go:160: Failed to list *api.LimitRange: client: etcd cluster is unavailable or misconfigured
E0306 01:08:48.053808       6 cacher.go:217] unexpected ListAndWatch error: pkg/storage/cacher.go:160: Failed to list *api.ResourceQuota: client: etcd cluster is unavailable or misconfigured
E0306 01:08:48.054264       6 cacher.go:217] unexpected ListAndWatch error: pkg/storage/cacher.go:160: Failed to list *api.Secret: client: etcd cluster is unavailable or misconfigured
E0306 01:08:48.054349       6 cacher.go:217] unexpected ListAndWatch error: pkg/storage/cacher.go:160: Failed to list *api.Namespace: client: etcd cluster is unavailable or misconfigured
E0306 01:08:48.054435       6 cache

Didn't find anything immediately suspicous in master kubelet or docker logs.

[dchen1107]

@lavalamp could you please take a look at this or delegate? Thanks!

[lavalamp]

@dchen1107 something bad happened to the node, maybe to docker. I see lots of

I0306 01:04:03.535184    3507 worker.go:161] Probe target container not found: etcd-server-e2e-gce-master-0-master_kube-system(047397a03841de24aac3acbd0cab7cfa) - etcd-container

messages on the master node's kubelet.log. Looks like it wasn't able to run much. etcd log is empty. Nothing obvious in docker log.

But anyway, the problem is clearly that etcd never started. Sorry, but I have to give this back to you :)

[dchen1107]

I looked at logs of master node.

• The etcd-server container didn't run because docker image isn't loaded. We are doing side-loading all docker images for master component, including etcd image.
• I checked docker log, it doesn't complain any failure of docker load operation.
• Also docker is not hung at all since all other images are properly loaded and containers are created and started.
• I checked salt related logs, and no errors.

I am wondering if etcd image tar file is copied to the master node properly through pr builder. Tried to reproduce it through kube-up but failed.

[dchen1107]

Together with @bprashanth, we double checked our master component tarball, which doesn't include etcd:2.2.1 image tar file. Looks like etcd:2.2.1 is the only image of master component which is pulled from gcr.io.

I checked kubelet.log on master node again, there are only image pulling event, no corresponding pulled event for etcd:2.2.1. Based on my understanding, the reason we want to keep image side-loading for master component is that gcr.io repository is not very reliable. We used to include etcd image in master component tarball too. cc/ @roberthbailey and @zmerlynn

[roberthbailey]

I don't recall a time when we were side-loading the etcd image. We load it from gcr.io in the release-1.1 branch and the release-1.0 branch and we haven't seen it have a significant impact on cluster creation reliability.

/cc @jlowdermilk

[dchen1107]

Do we have more instances with the same failure here? The problem is for this particular docker pull etcd:2.2.1 operation takes long time.

[dchen1107]

Since there is no instance with the same failure. I am closing this one for now.