Stop timeout isn't respected at shutdown/reboot

[mrunalp]

What happened:
Containers are getting terminated by systemd without respecting the terminationGracePeriodSeconds set in the pod yaml on reboot or shutdown of a node.

What you expected to happen:
terminationGracePeriodSeconds is respected by systemd when using systemd as the cgroup manager.

How to reproduce it (as minimally and precisely as possible):

1. Use systemd as the cgroup manager in your container runtime.
2. Create a pod yaml with terminationGracePeriodSeconds set to 120 seconds.
3. Reboot the node.
4. You will notice that the containers get SIGTERM, followed by systemd default stop timeout (typically 90 seconds) and then they are SIGKILLed.

Anything else we need to know?:
This can be fixed by passing the stop timeout to the containers as part of the CreateContainer CRI API. This will allow the container runtimes to set the systemd property for the scope to override the default stop timeout to the value set through terminationGracePeriodSeconds.
This needs changes across the stack as runc doesn't currently provide a way to set the TimeoutStopUSec for the systemd scope created for a container.
The behavior with cgroupfs cgroup manager will need further investigation.

Environment:

• Kubernetes version (use kubectl version): All versions.

[mrunalp]

cc: @derekwaynecarr @dchen1107

[derekwaynecarr]

while this impacts all pods, it is particularly an issue with static pods or daemon set backed pods which typically are not drained before a maintenance action.

/milestone v1.15

[yujuhong]

@derekwaynecarr IIUC, this is a feature request, since kubernetes has never supported this. Is there a reason why this is marked milestone 1.15 when we are already past enhancement freeze and almost reaching code freeze?

[Random-Liu]

I'm not sure whether we want the user configured grace period to block node reboot.

Do people only reboot node after draining pods?

If not, say I have a 10min grace period pod, will it block the node from reboot for 10min? Is it fair to the other pods which have 10s grace period? It seems that they may have a 10min downtime or rescheduled to other nodes unnecessarily?

[mrunalp]

We can restrict it similar to how privileged is restricted. This is useful for daemon sets and static pods that are system owned.

…

On May 29, 2019, at 5:32 PM, Lantao Liu ***@***.***> wrote: I'm not sure whether we want the user configured grace period to block node reboot. If I specify a 10min grace period, will it block the node from reboot for 10min? Is it fair to the other pods? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Random-Liu]

We can restrict it similar to how privileged is restricted. This is useful for daemon sets and static pods that are system owned.

If that is the case, it seems that we need to define Kubernetes api for that. Kubelet doesn't know whether a pod belongs to a daemonset or not.

BTW, maybe it makes sense to block node reboot for that long, I'm not sure, either. Just thinking out loud.

[mattjmcnaughton]

/sig node

[soggiest]

Is this issue release blocking or can we move it to 1.16?

[liggitt]

If this is not a regression in 1.15, I'd expect it can be moved

[soggiest]

/milestone v1.16

[josiahbjorgaard]

Hi all, code freeze for v1.16 is coming up on Aug. 29 (in just 6 days). This issue is set for milestone v1.16, will it be solved by then? It will need to have a PR merged beforehand. If not, we will remove the v1.16 milestone.

[liggitt]

I'm not aware of any activity on this issue targeting 1.16

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[dtolmachev]

@mrunalp
I've found this issue trying to fix the same problem as it is described in "How to reproduce it" section.

We run static pod with Docker container. Also we have kubelet and containerd on VM.

The problem: I want to make graceful shutdown with 60s duration for container. But currently, we have only 10 seconds. That's because Docker receives SIGTERM and it is interpreted as docker stop with default 10s timeout
It is possible to set --stop-timeout for docker run but we run container with kubelet and terminationGracePeriodSeconds in pod.yaml doesn't change the shutdown duration.
The pull-request in containerd project is still open.

Can I help with finishing this PR? What job needs to be done?

Thanks a lot for supporting Kubernetes and containerd!

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mrunalp]

Can this be reopened? I am opening a PR to address this.

[dims]

/reopen

[k8s-ci-robot]

@dims: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mrunalp]

@dims Thanks :)

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.