Reboot e2e test timeout because of slow docker startup

[bprashanth]

I've seen this fail on occasion on local e2e runs. It looks like what happens is:

• Reboot, kubelet logs show a bunch of binary looking output (I'm assuming from the reboot)

I0605 22:10:50.673994 @^ x 100

• Ssh fails for a while , e2e logs:

INFO: Error while issuing ssh command: when running echo b

• Ssh works, kubelet isn't up (10:50->11:50)

I0605 22:11:50.687609    2193 server.go:261] Using root directory: /var/lib/kubelet
I0605 22:11:50.688319    2193 manager.go:126] cAdvisor running in container: "/"

• Kubelet is up, docker isn't running

I0605 22:12:56.816171    2515 kubelet.go:2082] Query container info for pod "kibana-logging-v1-0cev8_default" failed with error (API error (500): engine is shutdown
...
E0605 22:14:15.630260    2515 kubelet.go:1644] Couldn't sync containers: dial unix /var/run/docker.sock: no such file or directory

• Test times out

Here's a gist of a segment of my logs:
https://gist.github.com/bprashanth/c90bd10b359851a57cb1

and heres describe events for the pod that the test complained about (elasticsearch-logging-v1-7w3d6):
https://gist.github.com/bprashanth/f380c2e80e5740c37520

@mforbes (just from git blame on test unsure if you care)

[bgrant0607]

See also #9062, #8962, #9117

[bprashanth]

I swear i searched for the error message and found nothing. I think the reboot test involves a whole lot of stuff and there's a good chance all 3 bugs are actually different, so retitling.

[dchen1107]

One way to speed up docker's startup is removing all containers before reboot for testing purpose. There is not much things we can do at node level.

[brendandburns]

The reboot test shows no failures in the last 30 runs due to this bug. I'm de-prioritizing back to P2, and if it continues to not be flaky, I'm going to kick it out of 1.0

[wojtek-t]

@jszczepkowski - can you please take a look?

[bprashanth]

Fyi I think reboot is part of the tests we're skipping (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/hack/jenkins/e2e.sh#L69), so if we don't think reboot is important/stable enough to run regularly, this probably isn't a 1.0 bug

[goltermann]

@quinton-hoole @fabioy Do we think this one is now fixed via #14772 , or is there more to do?

[ghost]

We're actually still seeing occasional timeouts since #14772 , so lets leave this one open to track those. e.g.
/job/kubernetes-e2e-gce-reboot/5589/
Reboot each node by triggering kernel panic and ensure they function upon restart
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:77
Oct 2 10:11:24.229: Node e2e-reboot-minion-6ab8 failed reboot test.

[davidopp]

Two failures today:
http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubernetes-e2e-gce-reboot/5653/
http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubernetes-e2e-gce-reboot/5654/

[ikehz]

These are still flaking occasionally:

• Reboot [Disruptive] each node by dropping all inbound packets for a while and ensure they function afterwards (Failed 1 times in the last 30 runs. Stability: 96 %) No flake or failure 3 min 56 sec Passed
• Reboot [Disruptive] each node by dropping all outbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) No flake or failure 3 min 57 sec Passed
• Reboot [Disruptive] each node by ordering clean reboot and ensure they function upon restart (No known failures. Stability 100 %) No flake or failure 1 min 58 sec Passed
• Reboot [Disruptive] each node by ordering unclean reboot and ensure they function upon restart (Failed 1 times in the last 30 runs. Stability: 96 %) No flake or failure 1 min 54 sec Passed
• Reboot [Disruptive] each node by switching off the network interface and ensure they function upon switch on (No known failures. Stability 100 %) No flake or failure 3 min 59 sec Passed
• Reboot [Disruptive] each node by triggering kernel panic and ensure they function upon restart (Failed 1 times in the last 30 runs. Stability: 96 %)No flake or failure 1 min 22 sec Passed

[bprashanth]

I think a lot of that is addressed in #19189 (comment)

[ikehz]

... except that both of those failures were during the TCP problems we saw yesterday, where a bunch of other tests were also failing.

[ikehz]

@dchen1107 Why did you assign this to me?

[ikehz]

Closing in favor of #19189.

[bprashanth]

Agree with the dupe. I don't think these were actually networking problems, or nodes not coming up. The cases I saw last week/start of this week were because of cluster addons/static pods not coming up. I might be wrong/taken a biased sampling of wrong logs.