kubelet won't retry PodSandbox creation for pods with restart policy "Never"

[yujuhong]

What happened:

If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever.

What you expected to happen:
Since the user/application containers were never created, kubelet should retry the PodSandbox creation.

The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user.

How to reproduce it (as minimally and precisely as possible):
I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations.

Anything else we need to know?:

Relevant code snippet:
https://github.com/kubernetes/kubernetes/blob/v1.16.0-alpha.0/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L474-L482

Environment:

• Kubernetes version (use kubectl version): Only checked the master branch (1.16), but all supported release branches should have the issue
• Cloud provider or hardware configuration: not relevant
• OS (e.g: cat /etc/os-release): not relevant
• Kernel (e.g. uname -a): not relevant
• Install tools:
• Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues.
• Others:

[mattjmcnaughton]

@yujuhong thanks for posting!

I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox.

I was thinking something similar to:

if createPodSandbox {
    sandboxUID, _ := m.getSandboxIDByPodUID(pod.UID, nil)
    if !shouldRestartOnFailure(pod) && sandboxUID != nil {
        ...
    }
    ...
}

Thoughts? Is there a simplex fix?

[edmorley]

Hi! Thank you for filing this - it matches the behaviour we've been seeing.

Am I correct that this is a regression from 5f473bc (#68980), and so only affects Kubernetes >=1.13 ?

[yujuhong]

I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox.

Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP.

Probably have to actually determine whether the user containers were created before. Need to check the code closer though.

[yujuhong]

Am I correct that this is a regression from 5f473bc (#68980), and so only affects Kubernetes >=1.13 ?

Thanks. That's probably correct. I saw this in happening in 1.13

[mattjmcnaughton]

On Wed, Jun 26, 2019 at 09:39:59AM -0700, Yu-Ju Hong wrote: > I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #79398 (comment)

Gotcha - thanks! Taking a look at your pr now :)