PreStop lifecycle hook feature

[torredil]

Is this a bug fix or adding new feature?

New feature.

What is this PR about? / Why do we need it?

• When a node is terminated, persistent workflows using EBS volumes can take 6+ minutes to start up again. This happens when a volume is not cleanly unmounted, which causes the Attach/Detach controller (in kube-controller-manager) to wait for 6 minutes before issuing a force detach and allowing the volume to be attached to another node. For context, the 6 minute interval is a non-configurable value used as a safety mechanism to handle uncertainty during workflow interruptions with the goal of preventing data corruption caused by forced detachments.
• The lifecycle hook aims to ensure that before the node (and the CSI driver node pod running on it) is shut down, all VolumeAttachment objects associated with that node are removed, indicating that all volumes have been successfully unmounted and detached. This way, we can be sure that all race conditions during the process are taken care of (node side and controller side).
• No unnecessary delay is added to the termination workflow, as the PreStop hook logic is only executed when the node is being drained (thus preventing delays in termination where the node pod is killed due to a rolling restart, or during driver upgrades, but the workload pods are expected to be running).
• By leveraging the PreStop hook, there's no need to rely on external tools or handlers like the AWS Node Termination Handler to prevent 6 minute delays.

What testing is done?

• Unit testing.
• Manual testing:

1. Setup dev cluster.

2. Build and deploy new driver image with changes introduced in this PR.

3. Apply dynamic provisioning manifests kubectl apply -f manifests:

persistentvolumeclaim/ebs-claim created
pod/app created
storageclass.storage.k8s.io/ebs-sc created

4. Verify VolumeAttachment object is present kubectl get volumeattachment:

kubectl get volumeattachment
NAME                                                                   ATTACHER          PV                                         NODE                  ATTACHED   AGE
csi-e34944e5bc014158229c0d1cb82f38d803f9f1eab0369280fddfb206c3d57b5d   ebs.csi.aws.com   pvc-90f38262-3d71-40be-beb4-553a773c74d9   i-01cf45161c87325e8   true       20s

5. Cordon node to simulate drain event, kubectl cordon <node-name>.

6. Describe node to confirm node is unschedulable. kubectl describe <node-name>.

Taints:             node.kubernetes.io/unschedulable:NoSchedule
Unschedulable:      true

Normal   NodeNotSchedulable       1s                 kubelet                Node i-01cf45161c87325e8 status is now: NodeNotSchedulable

7. Delete driver node pod on unschedulable node with associated VolumeAttachment object and confirm driver node pod is not taken down until either terminationGracePeriodSeconds elapses or VolumeAttachment object is removed.

[k8s-ci-robot]

@AndrewSirenko: changing LGTM is restricted to collaborators

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[torredil]

Pending rebase

[ConnorJC3]

some minor comments but largely lgtm

[ConnorJC3]

/test pull-aws-ebs-csi-driver-external-test-kustomize

[ConnorJC3]

/lgtm

[AndrewSirenko]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndrewSirenko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [AndrewSirenko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment