Iptables sporadically errs out on master

[bprashanth]

It is weird. This failure:

https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/22989/kubernetes-pull-build-test-e2e-gce/32669/artifacts/e2e-gce-builder-2-0-master/
#22989 (comment)

happened because the apiserver didn't come up in 300s, which happened because:

NodeHasSufficientDisk
I0315 14:32:55.787839    3394 kubelet.go:1137] Unable to register e2e-gce-builder-2-0-master with the apiserver: Post https://e2e-gce-builder-2-0-master/api/v1/nodes: dial tcp 10.240.0.2:443: connection refused
E0315 14:32:56.029664    3394 generic.go:195] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint
E0315 14:32:57.030062    3394 generic.go:195] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint
I0315 14:32:57.843326    3394 kubelet.go:2365] skipping pod synchronization - [Error on adding ip table rules: exit status 2 container runtime is down]

So essentially the Kubelet never setup cbr0 but did restart docker, which failed to startup because cbr0 doesn't exist.

[bprashanth]

So essentially the Kubelet never setup cbr0 but did restart docker, which failed to startup because cbr0 doesn't exist.

Actually how did this happen? it must've been a spurious docker restart (because we only pkill docker after configuring iptables rules, which is failing).

Also I noticed it was failing in the same way on all nodes, so not sure what's going on at this point. Maybe it was a real kubelet bug that we fixed (but iptables exiting with 2 is indicative of, eg, wrong formatting of rules and we haven't touched the rule in question in a while, right?

kubernetes/pkg/kubelet/container_bridge.go

Line 145 in dae5ac4

func ensureIPTablesMasqRule(nonMasqueradeCIDR string) error {

)