Kubelet restart terminates graceful termination

[lbernail]

If the kubelet is restarted while some containers are in graceful termination and waiting for the period to expire, waitContainerStop will return on context cancellation and the task will be immediately killed.

Steps to reproduce:

• create a pod with a long terminationGracePeriod
• delete the pod, it will enter the TERMINATING phase
• restart (or stop) the kubelet => the associated container will immediately be killed and the pod deleted

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sig
  labels:
    app: sig
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sig
  template:
    metadata:
      labels:
        app: sig
    spec:
      tolerations:
      - operator: Exists
      containers:
      - name: sig
        image: lbernail/sig:0.1
        imagePullPolicy: Always
      terminationGracePeriodSeconds: 3600

(lbernail/sig:0.1 simply ignores SIGTERM)

After deleting the pod, we get these logs:

containerd[951]: time="2019-03-25T12:05:25Z" level=info msg="StopContainer for "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" with timeout 3600 (s)"
containerd[951]: time="2019-03-25T12:05:25Z" level=info msg="Stop container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" with signal terminated"

If we then restart/stop the kubelet, we immediately get:

containerd[951]: time="2019-03-25T12:05:51Z" level=error msg="Stop container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" timed out" error="wait container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" is cancelled"
containerd[951]: time="2019-03-25T12:05:51Z" level=info msg="Kill container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4""
containerd[951]: time="2019-03-25T12:05:51Z" level=error msg="StopContainer for "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" failed" error="an error occurs during waiting for container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" to stop: wait container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" is cancelled"
containerd[951]: time="2019-03-25T12:05:51Z" level=info msg="Finish piping stdout of container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4""
containerd[951]: time="2019-03-25T12:05:51Z" level=info msg="Finish piping stderr of container "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4""
containerd[951]: time="2019-03-25T12:05:51Z" level=info msg="shim reaped" id=2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4
containerd[951]: time="2019-03-25T12:09:50Z" level=info msg="Container to stop "2d533fb60ec326c7b46c76c9b11c1c9e09fe694f59a3bc5baa8f46e114749ab4" is not running, current state "CONTAINER_EXITED""

cc @Random-Liu

[Random-Liu]

Very good catch. At least this proves that we are handling context cancellation properly, and there is no request handler goroutine leakage. :)

What is the expected behavior here? Probably abort the stop waiting and do not continue doing SIGKILL?

[lbernail]

Would it be possible to continue waiting and kill the container if we reach timeout (ignore the context cancellation)?

[Random-Liu]

@lbernail That can potentially cause goroutine leakage. I don't think we should do that.

We cancel the context on the kubelet side, if stop container timeout. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/remote/remote_runtime.go#L237
That is just to avoid goroutine leakage.

[lbernail]

The containerd goroutine waiting for the container to stop should not leak because it will be killed after timeout anyway? (I may totally be missing something here)
Aborting the stop and not killing would definitely be a better behavior. On restart the kubelet should trigger a new StopContainer right?

[Random-Liu]

The containerd goroutine waiting for the container to stop should not leak because it will be killed after timeout anyway? (I may totally be missing something here)

No one kills it, we have to rely on context cancellation to stop the goroutine ourselves.

On restart the kubelet should trigger a new StopContainer right?

Yes.