The driver does not properly release file handles for decomissioned pods

[david-garcia-garcia]

What happened:

If a pod is manipulating files on a FileShare a handle is opened.

You can retrieve them using Get-AzStorageFileHandle

When the pod is forcefully closed, the handle is not released. Because the actual SMB handle belongs to the K8S node the handle is left dangling forever until the node is replaced.

This is causing us problems with locked files that are never released.

Not sure if anything can be done at the CSI driver level to try to cleanup dead handles belonging to non existent pods or properly close them when the pod is killed.

What you expected to happen:

Handles acquired from a pod are released when the pod is killed.

How to reproduce it:

Store a compressed (big enough so that it takes time to decompress) file on a file share. Start decompression from a pod. Kill the pod.

Anything else we need to know?:

Environment:

[andyzhangx]

CSI driver only deals with mount/unmount, the file handle is opened by application pods, there is nothing CSI driver could do to avoid such issue.

[goosmurf]

@david-garcia-garcia are your pods able to exit cleanly?

We have seen a possibly related issue where a pod is unable to terminate cleanly because the application gets stuck on a CIFS operation (against an Azure File Share). The app exits as cleanly as it can and the app process itself is Zombied, but threads remain in Disk sleep state, with stack traces that are stuck on a CIFS operation. So far we have not found a way to clean this up.

e.g. stack

[<0>] wait_on_page_bit_common+0xff/0x3c0
[<0>] wait_on_page_bit+0x3f/0x50
[<0>] wait_on_page_writeback+0x26/0x80
[<0>] cifs_writepages+0x5b1/0xd70 [cifs]
[<0>] do_writepages+0xcc/0x1f0
[<0>] filemap_fdatawrite_wbc+0x81/0xd0
[<0>] filemap_write_and_wait_range+0x72/0xe0
[<0>] cifs_flush+0x67/0x100 [cifs]
[<0>] filp_close+0x37/0x70
[<0>] close_fd+0x38/0x60
[<0>] __x64_sys_close+0x11/0x50
[<0>] x64_sys_call+0x1de6/0x1fa0
[<0>] do_syscall_64+0x56/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6

This is a rare occurrence.

So far when this has happened and I have listed the open handles from the Azure File Share side, then done an az storage file show on each file with an open handle, there are files that are stuck in DeletePending state. I don't know that the DeletePending is related or coincidental.