Cleanup webhooks using pod termination

[realshuting]

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks

[shivdudhani]

PreStop
Called before the container is terminated.
Termination of Pods:
When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container. Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.

By default, all deletes are graceful within 30 seconds. This 30 seconds time is all time that methods PreStop hook, TERM and KILL have to complete by. If the user explicitly sets graceful period to 0 and force, then the API server does not wait for confirmation before removing the resource from API.

https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html provides some explanation for the graceful shutdown of pods.
It states 2 points:

• make sure the base image is forwarding signals to the application. [Requires testing]
• some applications handle SIGTERM differently, for example nginx on recieving SIGTERM kills the process. But uses nginx -s quit for graceful shutdown, so here preStop hook can send the quit signals got graceful shutdown.

In kyverno we use a channel to be notified when signals are passed, and this channel is used to start cleanUp, and another notification channel to wait till cleanUp is complete(i.e. webhookconfigurations are deleted).
With nirmata@439c64b we have a helper function to mock signals, so this can be used to test the cleanUp.

Comments:
I think using a preStop hook won't help with the cleanup, but we can test further.
It might be helpful to have a separate goroutine that's responsible for cleanup and make sure it's small and fast.
Currently cleanup routine is serial now,

• firstly resource mutating webhook configuration
• secondly, policy validating webhook configuration
• thirdly policy mutating webhook configuration

So the graceful shutdown time for each step is 30 seconds making the worst-case time to 90 seconds. With the pod only having 30 seconds to complete, it won't finish. Cleanup up parallelly would benefit theoretically.

@JimBugwadia @realshuting
Let me know you thoughts on the above!

[shivdudhani]

by default http.Server waits indefinitely for connections to return to idle and then shuts down. Then we clean up the webhookconfigurations.
https://github.com/nirmata/kyverno/blob/bdb677abf6b75b4c93a3c34490acd60ba677b909/pkg/webhooks/server.go#L199-L207

Proposal:

• Adjust the default deadline for context to 5*seconds, any request being processed is cleaned up.
• Call cleanup webhookconfigurations before http server shutdown and clean up each resource in a separate goroutine with a wait group to wait for completion. After the cleanup is done we close the notification channel that waits on in the main goroutine. This will us to log if the completion was complete or not(example log: successful shutdown of kyverno controller).