isDeviceOpened check for cinder could use invalid device path

[gnufied]

We fixed invalid device path in bunch of places via - #33270 but isDeviceOpen call still could use invalid device path.

That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted.

/sig storage
/sig openstack

/assign

[gnufied]

Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now.

/close

[gnufied]

Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now.

/close

[gnufied]

I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.

[gnufied]

I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.

[msau42]

How come the check can "sometimes" use invalid device path?

[msau42]

How come the check can "sometimes" use invalid device path?

[gnufied]

@msau42 So on Cinder for example - Attach call would return something like /dev/sdb as attach device path. But if you see the linked PR(#33270), WaitForAttach will do a /dev/disk/by-id and use that device path for global mount. In general it is always a good idea to use device path returned by WaitForAttach as single source of truth.

This applies to something like EBS with NVMe too where device is discovered via WaitForAttach call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).

[gnufied]

@msau42 So on Cinder for example - Attach call would return something like /dev/sdb as attach device path. But if you see the linked PR(#33270), WaitForAttach will do a /dev/disk/by-id and use that device path for global mount. In general it is always a good idea to use device path returned by WaitForAttach as single source of truth.

This applies to something like EBS with NVMe too where device is discovered via WaitForAttach call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).

[gnufied]

So on EBS with NVMe for example - isDeviceOpen check will always fail (but still return success) because it will use something like /dev/xvdba as device path which does not actually exist on the node.

[gnufied]

So on EBS with NVMe for example - isDeviceOpen check will always fail (but still return success) because it will use something like /dev/xvdba as device path which does not actually exist on the node.

[msau42]

You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?

[msau42]

You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?

[gnufied]

@msau42 doesn't the linked PR #33270 make it clear?

We can't rely on the device name provided by Cinder, and thus must perform
detection based on the drive serial number (aka It's cinder ID) on the
kubelet itself.

As for why - Cinder sometimes returns wrong device path on AttachDevice call, I don't know. But it happens and we have worked around it.

[gnufied]

@msau42 doesn't the linked PR #33270 make it clear?

We can't rely on the device name provided by Cinder, and thus must perform
detection based on the drive serial number (aka It's cinder ID) on the
kubelet itself.

As for why - Cinder sometimes returns wrong device path on AttachDevice call, I don't know. But it happens and we have worked around it.

[gnufied]

Fixed by #68674

[gnufied]

Fixed by #68674