Resize volume fails after DRS migrates volume in datastore cluster

[harikrishna-patnala]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VMware, DatastoreCluster

CLOUDSTACK VERSION

main

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

In VMware datastore cluster, if DRS triggers a volume migration and if the volume path has changed then the subsequent resize volume operation in CloudStack fails because the update volume path is not there with CloudStack.

Workaround is to stop start the VM or detach and attach the volume.

STEPS TO REPRODUCE

1. Create a VM with datadisk in CloudStack, make sure volumes created in datastore cluster
2. Let DRS triggers migration of volumes within datastore cluster
3. make sure of the usecase that volume path (name) has changed (not just the datastore)
4. Try resize volume operation from CloudStack

EXPECTED RESULTS

Volume resize should be successful

ACTUAL RESULTS

Volume resize fails with error "No such disk device"

[mdesaive]

Hi Harikrishna,

thank you for addressing this bug and creating this ticket.

Only for my understanding and to make sure we address the problem properly:

You had been aware that there are inconsistencies to be expected with the path and chaininfo field after storage VMotion.

There was a bug starting VMs with volumes with such inconsistencies. (#5091) This should be fixed with 4.15.1

Do I understand you correctly that the approach to fix this bug will be to add code to the volume resize API call, that corrects eventually inconsistent data?

If so, what about volume migrations? Should we test, if there is also a problem when the data is inconsistent, or has that already been addressed in some other issue?

Kind regards,

Melanie

[mdesaive]

And concerning your question about when the problem occurs:

Both for manual storage VMotion and DRS VMotion VMWare changes the volume name from the UUID type to some string consisting of VM name and count, which breaks consistency with the CloudStack database.

[mdesaive]

The bug occurs in the following situation:

If a volume is created by CloudStack and not ingested from VMWare the path is formed as an UUID. VMWare is actually working with a determined value formed of the VM name and a suffix for counting as volume name.

With a live storage migration on VMWare level this volume name is changed from the UUID type to the VM name type and the CloudStack database is not updated.

Live storage migrations from VMWare/DRS seem to not change the volume name, when the volume already is stored in the VMWare style. This happens for the volumes, that where ingested with taking before unmanaged VMs into CloudStack management. These are "safe".

[mdesaive]

We seemed to have found a valid temporary workaround for us:

After a volume was attached, we perform a volume live migration with CloudStack.
After this had succeeded the volume path in VMWare and the path and chaininfo entries in the CloudStack database are consistently changed to the VMWare naming scheme and afterwards no longer changed through storage migrations triggered from VMWare.

[harikrishna-patnala]

Thanks for the details Melanie. To answer your question, yes fix is during volume resize API call MS check if it has the right data, if not, updates the DB accordingly.

Yes the workaround you mentioned is also fine, you can also stop start the VM or detach and attach the disk to get the proper data into CS.

[mdesaive]

What will happen, when I trigger a volume live migration with inconsistent data in the CS database for path or chaininfo?

[harikrishna-patnala]

MS tries to fetch volume details with old data if not it fetches proper data from vCenter and updates CS during migration operation itself. The same thing we will do it for resize operation