kubelet: pod should be terminated immediately once preStop complete

[njuicsgz]

• Use case
  I have a RC with 5 pods. preStop.sh is defined to process all requests before shutdown the service in container. preStop.sh takes almost 2min generally, or 5min at most. So I should set "terminationGracePeriodSeconds": 300 for a graceful termination.
• Current behavior
  I MUST wait for 5min no matter whether the preStop.sh is completed. when I use rolling update for all 5 pods, it takes 5min*5 = 25min.
• Expected behavior
  Terminate pod immediately if preStop is configured and completed. So in most cases, the rolling update will just spend 10min. In my opinion, preStop is used to graceful termination, so there is no need to wait a fixed terminationGracePeriodSeconds time.
• fix proposal
  Just set 'gracePeriod = 0'.
  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/manager.go?#L1345

I will send a PR if it is OK.

[yujuhong]

PreStop hooks are indeed mostly used for shutting down the applications gracefully, but I'm not sure whether there are other use cases out there. Changing this could potentially break those users.

/cc @kubernetes/sig-node @smarterclayton

[smarterclayton]

You shouldn't have to wait five minutes - are you eating SIGTERM in your main process? If you handle SIGTERM correctly then when Prestop returns you'll be signaled immediately. So having to wait five minutes sounds like either Prestop isn't exiting correctly or your process isn't listening for term.

Prestop isn't intended to allow you to escape the graceful shutdown quota. It's a contract with the rest of the system.

[njuicsgz]

@smarterclayton I have not eaten the SIGTERM, but can not terminate pod after PreStop return.

$ cat nginx.term.json
{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"name": "nginx",
"namespace": "default"
},
"spec": {
"replicas": 2,
"selector": {
"name": "nginx",
"version": "latest"
},
"template": {
"metadata": {
"labels": {
"name": "nginx",
"version": "latest"
},
"namespace": "default"
},
"spec": {
"terminationGracePeriodSeconds": 300,
"containers": [
{
"image": "nginx",
"name": "nginx",
"command": [
"/bin/bash",
"-c",
"echo $HOSTNAME > /usr/share/nginx/html/index.html && service nginx start && while (true); do sleep 1; done"
],
"ports": [],
"lifecycle": {
"preStop": {
"exec": {
"command": [
"bash",
"-c",
"sleep 6"
]
}
}
},
"resources": {
"limits": {
"memory": "128Mi"
}
}
}
]
}
}
}
}

$kubectl create -f nginx.term.json
$ kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-m5pbz 1/1 Running 0 5m
nginx-zs43i 1/1 Running 1 6m

$kubectl delete po nginx-m5pbz
pod "nginx-m5pbz" deleted

$# kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-5fxr0 1/1 Running 0 1m
nginx-m5pbz 1/1 Terminating 0 7m ===> already 1m, it should be terminated in 8s(6+2), as you said
nginx-zs43i 1/1 Running 1 8m

[smarterclayton]

I don't think "service nginx start" in bash will proxy the SIGTERM we send to bash (which is the parent process). You'll need to add a trap handler to your bash script in command and "service nginx stop".

[njuicsgz]

@smarterclayton As your suggestion, I changed my RC profile, add it works well. Thanks.

$ cat Dockerfile
FROM ubuntu:14.04
ADD /bootstrap.sh /
CMD ["/bootstrap.sh"]

$ cat bootstrap.sh
#!/bin/bash
trap 'my_exit; exit' SIGTERM
count=0

my_exit()
{
echo "you hit SIGTERM, now exiting.."
}

while :
do
sleep 1
count=$(expr $count + 1)
echo $count
done

$ cat term.json
{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"name": "term",
"namespace": "default"
},
"spec": {
"replicas": 2,
"selector": {
"name": "term",
"version": "latest"
},
"template": {
"metadata": {
"labels": {
"name": "term",
"version": "latest"
},
"namespace": "default"
},
"spec": {
"terminationGracePeriodSeconds": 60,
"containers": [
{
"image": "xa.repo.ndp.com:5000/paas/busy-term",
"name": "term",
"ports": [],
"lifecycle": {
"preStop": {
"exec": {
"command": [
"bash",
"-c",
"sleep 10"
]
}
}
},
"resources": {
"limits": {
"memory": "16Mi"
}
}
}
]
}
}
}
}

[prathameshd9]

@smarterclayton As your suggestion, I changed my RC profile, add it works well. Thanks.

$ cat Dockerfile
FROM ubuntu:14.04
ADD /bootstrap.sh /
CMD ["/bootstrap.sh"]

$ cat bootstrap.sh
#!/bin/bash
trap 'my_exit; exit' SIGTERM
count=0

my_exit()
{
echo "you hit SIGTERM, now exiting.."
}

while :
do
sleep 1
count=$(expr $count + 1)
echo $count
done

$ cat term.json
{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"name": "term",
"namespace": "default"
},
"spec": {
"replicas": 2,
"selector": {
"name": "term",
"version": "latest"
},
"template": {
"metadata": {
"labels": {
"name": "term",
"version": "latest"
},
"namespace": "default"
},
"spec": {
"terminationGracePeriodSeconds": 60,
"containers": [
{
"image": "xa.repo.ndp.com:5000/paas/busy-term",
"name": "term",
"ports": [],
"lifecycle": {
"preStop": {
"exec": {
"command": [
"bash",
"-c",
"sleep 10"
]
}
}
},
"resources": {
"limits": {
"memory": "16Mi"
}
}
}
]
}
}
}
}

I am not sure how this works ??! Is the trap handled in the preStop(within the commands/script running in preStop) or outside preStop ? Please help !