DRA: interaction with unexpected node shutdown KEP

[pohly]

We need to explore how DRA interacts with https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown.

One likely outcome is that the DRA helper code watches for nodes with the special node.kubernetes.io/out-of-service=nodeshutdown:NoExecute label and helps DRA drivers clean up.

/sig node

[bart0sh]

/assign

[bart0sh]

/triage important-soon
as this is a DRA Beta criteria and DRA is going to be graduated to Beta in 1.29

[k8s-ci-robot]

@bart0sh: The label(s) triage/important-soon cannot be applied, because the repository doesn't have them.

In response to this:

/triage important-soon
as this is a DRA Beta criteria and DRA is going to be graduated to Beta in 1.29

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bart0sh]

/triage accepted

[pohly]

Here's a test that was written for volumes: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/non_graceful_node_shutdown.go

[bart0sh]

Thank you. I’ve found it last week. Unfortunately kind image doesn’t have ssh, so stopping kubelet is not possible the way this test does. I’ve found another solution. From: Patrick Ohly ***@***.***> Date: Tuesday, 26. September 2023 at 8.46 To: kubernetes/kubernetes ***@***.***> Cc: Bartosh, Eduard ***@***.***>, Mention ***@***.***> Subject: Re: [kubernetes/kubernetes] DRA: interaction with unexpected node shutdown KEP (Issue #120421) Here's a test that was written for volumes: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/non_graceful_node_shutdown.go — Reply to this email directly, view it on GitHub<#120421 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADILHT4PO3BXDU3HGDK4KVLX4JT4BANCNFSM6AAAAAA4LSFNZQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> --------------------------------------------------------------------- Intel Finland Oy Registered Address: PL 281, 00181 Helsinki Business Identity Code: 0357606 - 4 Domiciled in Helsinki This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

[bart0sh]

I was able to simulate non-graceful node shutdown by running docker stop <node name>.
It turned out that after adding node.kubernetes.io/out-of-service=nodeshutdown:NoExecute tent, all node pods are deleted by the GC controller and the resource claim controller successfully deallocated the resource.

So, it looks like a non-graceful shutdown is handled correctly and no code changes are required.

However, the fact that Deallocate can potentially be called without calling NodeUnprepareResource should be explained in the DRA KEP, so that DRA plugin authors will be aware of this.

I'll add an explanation of handling non-graceful node shoutdown to the DRA KEP and add a correspondent e2e test case to the DRA e2e test suite.

[bart0sh]

KEP update PR: kubernetes/enhancements#4260
E2E PR: #120965