ReplicationController + crashlooping or invalid docker image = bad times

[lavalamp]

ReplicationController has no idea when the pods it is making will deterministicly fail. This can rapidly fill your cluster with pod objects.

Mitigation: throttle the rate at which rep. ctlrs. can make new pods.

Real fix? Make replication controller watch events & stop after N tries failed without any successes?

[bgrant0607]

See also #941 -- Kubelet has the same problem at the container level.

I'll leave this open to address my comment on #2491 -- gracefully handle pods that don't schedule.

[lavalamp]

To fix your cluster, not that anyone has ever done this:

cluster/kubectl.sh get --output=template --template="{{range .items}}{{.id}}{{\"\n\"}}{{end}}" pods | xargs -l1 cluster/kubectl.sh delete pod

[rawlingsj]

I've been caught out by this recently too. Having configuration for max number of tries on a replication controller would be great.

[davidopp]

I am confused about this issue. IIUC replication controller only works with pods that specify RestartPolicy = Always. So wouldn't a fix for this (limit # retries, or rate limit retries) need to go into the Kubelet (or whatever is restarting the container -- is it Docker or the Kubelet)? I don't see how replication controller can help. It seems replication controller would only get involved if the pod needs to move off of that machine, but that doesn't seem like the case you're talking about here.

[lavalamp]

@davidopp The problem is in how we report the pod's status; we report it as "failed" when kubelet is actually in the process of restarting it. This causes rep. ctrlr to make more pods, instead of waiting. So the bug here is in the assignment of status. Er, condition. Whatever we're calling it these days.

[bgrant0607]

ReplicationController shouldn't stop permanently, it should back off. It can be very hard to distinguish temporary from permanent failures, and we don't want users to need to create replication controller controllers. The same applies for Kubelet.

We should find a way to facilitate implementation of more sophisticated policies by users.

[lavalamp]

To recap.

While kubelet is restarting a pod, apiserver will list its status as "Failed", which will cause replication controller to make additional pods.

Action item here is to make apiserver give a reasonable amount of time (forever?) to kubelet to perform as many restarts as it wants, and during that time the pod should count as Running (if it ever ran) or Pending (if it never successfully started).

[lavalamp]

(Eventually we want to push pod status generation down into kubelet but I think it's worthwhile making this intermediate fix because this really hoses your cluster when it happens to you.)

[thockin]

I don't think it is about "reasonable amount of time" but having enough
information to make the correct observation.

On Tue, Jan 6, 2015 at 4:40 PM, Daniel Smith notifications@github.com
wrote:

To recap.

While kubelet is restarting a pod, apiserver will list its status as
"Failed", which will cause replication controller to make additional pods.

Action item here is to make apiserver give a reasonable amount of time
(forever?) to kubelet to perform as many restarts as it wants, and during
that time the pod should count as Running (if it ever ran) or Pending (if
it never successfully started).

Reply to this email directly or view it on GitHub
#2529 (comment)
.

[dchen1107]

@lavalamp #2999 was merged a while back, and with that change, now PodCondition / PodStatus are determined by both ContainerStatus and RestartPolicy.

[lavalamp]

Dawn is right, this bug seems to be fixed and should probably be closed.

[bgrant0607]

There's still the broader issue of making replication controller do something useful (e.g., raising events, including number of pending pods in status) in this scenario.

[liggitt]

xref #76370