Volume attachment not cleared before worker delete

[linwalth]

How to categorize this issue?

Our Gardener Cluster running on OpenStack runs in reconciliation issues because it is not detaching volumes from shoot workers.

/area storage
/topology shoot
/kind bug

What happened:
When an instance in a shoot is deleted, the volume detachment is not finalized before deletion of the shoot worker node. Thus, in some cases, the volume will still be counted as attached and the shoot cluster is thus running into reconciliation errors.

What you expected to happen:
An instance will only ever be deleted after all cinder volumes are functionally detached from the Instance.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Gardener version: 1.76.3
• Kubernetes version (use kubectl version): 1.24.5
• Cloud provider or hardware configuration: OpenStack

[kon-angelo]

Potentially this could be an issue for https://github.com/gardener/gardener-extension-provider-openstack. But can you actually go into more detail (or potentially a step-by-step description) of what is happening ? I find the description somewhat too terse.

On instance (by instance I assume an OS server/node), the MCM should try to drain the node before attempting to delete it. That should take care of moving most workloads out of the node, including moving the volumes after their pods have been scheduled elsewhere. Did you see issues during this process ?
Were the volumeattachments not being deleted, or maybe where there issues with CSI preventing the detach ?

[linwalth]

I am sorry, i cannot provide more context, as this ticket is already me describing a blackbox ;)

Given that the mechanism you described should take care of detaching/migrating all existing resources attached to and scheduled on that node I think this problem might be more on openstack side than on gardener side...

This is something that usually happens during and after Openstack upgrades.