Feat/add vdisk UUID to list volume #5848

DK101010 · 2022-01-11T08:05:14Z

Each VDisk on hard disk have got a uuid which starts with 600xx. Administrators needs this uuid often to identifies the right VDisk.

This PR get the vdisk uuid from vcenter and store it into the database as external_uuid in volume table. Also extend the listVolume response to add external_uuid.

Only new volumes are affected by this implementation.

Two sql update steps are necessary:

1

ALTER TABLE volumes ADD COLUMN external_uuid varchar(40) DEFAULT null;

2

DROP VIEW IF EXISTS `cloud`.`volume_view`;
CREATE VIEW `cloud`.`volume_view` AS
    SELECT
        volumes.id,
        volumes.uuid,
        volumes.name,
        volumes.device_id,
        volumes.volume_type,
        volumes.provisioning_type,
        volumes.size,
        volumes.min_iops,
        volumes.max_iops,
        volumes.created,
        volumes.state,
        volumes.attached,
        volumes.removed,
        volumes.display_volume,
        volumes.format,
        volumes.path,
        volumes.chain_info,
        volumes.external_uuid,
        account.id account_id,
        account.uuid account_uuid,
        account.account_name account_name,
        account.type account_type,
        domain.id domain_id,
        domain.uuid domain_uuid,
        domain.name domain_name,
        domain.path domain_path,
        projects.id project_id,
        projects.uuid project_uuid,
        projects.name project_name,
        data_center.id data_center_id,
        data_center.uuid data_center_uuid,
        data_center.name data_center_name,
        data_center.networktype data_center_type,
        vm_instance.id vm_id,
        vm_instance.uuid vm_uuid,
        vm_instance.name vm_name,
        vm_instance.state vm_state,
        vm_instance.vm_type,
        user_vm.display_name vm_display_name,
        volume_store_ref.size volume_store_size,
        volume_store_ref.download_pct,
        volume_store_ref.download_state,
        volume_store_ref.error_str,
        volume_store_ref.created created_on_store,
        disk_offering.id disk_offering_id,
        disk_offering.uuid disk_offering_uuid,
        disk_offering.name disk_offering_name,
        disk_offering.display_text disk_offering_display_text,
        disk_offering.use_local_storage,
        disk_offering.system_use,
        disk_offering.bytes_read_rate,
        disk_offering.bytes_write_rate,
        disk_offering.iops_read_rate,
        disk_offering.iops_write_rate,
        disk_offering.cache_mode,
        storage_pool.id pool_id,
        storage_pool.uuid pool_uuid,
        storage_pool.name pool_name,
        cluster.id cluster_id,
        cluster.name cluster_name,
        cluster.uuid cluster_uuid,
        cluster.hypervisor_type,
        vm_template.id template_id,
        vm_template.uuid template_uuid,
        vm_template.extractable,
        vm_template.type template_type,
        vm_template.name template_name,
        vm_template.display_text template_display_text,
        iso.id iso_id,
        iso.uuid iso_uuid,
        iso.name iso_name,
        iso.display_text iso_display_text,
        resource_tags.id tag_id,
        resource_tags.uuid tag_uuid,
        resource_tags.key tag_key,
        resource_tags.value tag_value,
        resource_tags.domain_id tag_domain_id,
        resource_tags.account_id tag_account_id,
        resource_tags.resource_id tag_resource_id,
        resource_tags.resource_uuid tag_resource_uuid,
        resource_tags.resource_type tag_resource_type,
        resource_tags.customer tag_customer,
        async_job.id job_id,
        async_job.uuid job_uuid,
        async_job.job_status job_status,
        async_job.account_id job_account_id,
        host_pod_ref.id pod_id,
        host_pod_ref.uuid pod_uuid,
        host_pod_ref.name pod_name,
        resource_tag_account.account_name tag_account_name,
        resource_tag_domain.uuid tag_domain_uuid,
        resource_tag_domain.name tag_domain_name
    from
        `cloud`.`volumes`
            inner join
        `cloud`.`account` ON volumes.account_id = account.id
            inner join
        `cloud`.`domain` ON volumes.domain_id = domain.id
            left join
        `cloud`.`projects` ON projects.project_account_id = account.id
            left join
        `cloud`.`data_center` ON volumes.data_center_id = data_center.id
            left join
        `cloud`.`vm_instance` ON volumes.instance_id = vm_instance.id
            left join
        `cloud`.`user_vm` ON user_vm.id = vm_instance.id
            left join
        `cloud`.`volume_store_ref` ON volumes.id = volume_store_ref.volume_id
            left join
        `cloud`.`disk_offering` ON volumes.disk_offering_id = disk_offering.id
            left join
        `cloud`.`storage_pool` ON volumes.pool_id = storage_pool.id
            left join
        `cloud`.`host_pod_ref` ON storage_pool.pod_id = host_pod_ref.id
            left join
        `cloud`.`cluster` ON storage_pool.cluster_id = cluster.id
            left join
        `cloud`.`vm_template` ON volumes.template_id = vm_template.id
            left join
        `cloud`.`vm_template` iso ON iso.id = volumes.iso_id
            left join
        `cloud`.`resource_tags` ON resource_tags.resource_id = volumes.id
            and resource_tags.resource_type = 'Volume'
            left join
        `cloud`.`async_job` ON async_job.instance_id = volumes.id
            and async_job.instance_type = 'Volume'
            and async_job.job_status = 0
            left join
        `cloud`.`account` resource_tag_account ON resource_tag_account.id = resource_tags.account_id
            left join
        `cloud`.`domain` resource_tag_domain ON resource_tag_domain.id = resource_tags.domain_id;

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested in own VMware environment.

sureshanaparti · 2022-01-11T08:28:08Z

@blueorangutan package

blueorangutan · 2022-01-11T08:29:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-01-11T09:36:51Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2174

sureshanaparti · 2022-01-11T12:25:36Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2022-01-11T12:26:03Z

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2022-01-11T13:05:32Z

Trillian Build Failed (tid-2860)

DaanHoogland · 2022-01-12T10:03:31Z

@DK101010 , if these sql are needed they should be added to the upgrade script, in this case in schema-41610to41700.sql

sureshanaparti · 2022-01-13T06:47:03Z

@blueorangutan package

blueorangutan · 2022-01-13T06:48:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-01-13T06:49:26Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2190

vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java

nvazquez · 2022-02-06T02:48:46Z

@blueorangutan package

blueorangutan · 2022-02-06T02:49:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-06T02:50:19Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2470

nvazquez · 2022-02-07T03:12:31Z

Hi @DK101010 can you please fix the open comments? Also, it seems there is an issue with packaging

DK101010 · 2022-02-09T09:31:38Z

Hi @DK101010 can you please fix the open comments? Also, it seems there is an issue with packaging

@nvazquez Sorry for delay, problems with my e-mail notifications. Regard packaging sounds strange that this change have an impact. Where can I check the packaging results?

nvazquez · 2022-02-09T16:57:12Z

@DK101010 I verified there are some conflicts that needs to be fixed, please rebase the current main branch

INFO: Executing the merge now.. Git output below:
INFO: ***********************************************************************************
From https://github.com/apache/cloudstack
 * [new ref]         refs/pull/5848/head -> pr/5848
Auto-merging vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java
Auto-merging server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
Auto-merging server/src/main/java/com/cloud/api/query/dao/VolumeJoinDaoImpl.java
Auto-merging plugins/hypervisors/vmware/src/main/java/com/cloud/storage/resource/VmwareStorageProcessor.java
Auto-merging engine/storage/volume/src/main/java/org/apache/cloudstack/storage/volume/VolumeObject.java
Auto-merging engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql
CONFLICT (add/add): Merge conflict in engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql
Auto-merging api/src/main/java/org/apache/cloudstack/api/ApiConstants.java
Auto-merging api/src/main/java/com/cloud/storage/Volume.java
Automatic merge failed; fix conflicts and then commit the result.
ERROR: Merge failed, aborting.

Build step 'Execute shell' marked build as failure
New run name is '#3540-5848'
Stopping all containers
Finished: FAILURE

DK101010 · 2022-02-10T07:45:40Z

@DK101010 I verified there are some conflicts that needs to be fixed, please rebase the current main branch

INFO: Executing the merge now.. Git output below:
INFO: ***********************************************************************************
From https://github.com/apache/cloudstack
 * [new ref]         refs/pull/5848/head -> pr/5848
Auto-merging vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java
Auto-merging server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
Auto-merging server/src/main/java/com/cloud/api/query/dao/VolumeJoinDaoImpl.java
Auto-merging plugins/hypervisors/vmware/src/main/java/com/cloud/storage/resource/VmwareStorageProcessor.java
Auto-merging engine/storage/volume/src/main/java/org/apache/cloudstack/storage/volume/VolumeObject.java
Auto-merging engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql
CONFLICT (add/add): Merge conflict in engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql
Auto-merging api/src/main/java/org/apache/cloudstack/api/ApiConstants.java
Auto-merging api/src/main/java/com/cloud/storage/Volume.java
Automatic merge failed; fix conflicts and then commit the result.
ERROR: Merge failed, aborting.

Build step 'Execute shell' marked build as failure
New run name is '#3540-5848'
Stopping all containers
Finished: FAILURE

Thanks, i will do this.

blueorangutan · 2022-04-06T12:10:40Z

Trillian Build Failed (tid-3830)

nvazquez · 2022-04-07T13:28:54Z

@blueorangutan test centos7 vmware-67u3 keepEnv

blueorangutan · 2022-04-07T13:30:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2022-04-07T13:47:43Z

Trillian Build Failed (tid-3839)

nvazquez · 2022-04-07T13:58:16Z

engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql

@@ -272,6 +291,7 @@ CREATE VIEW `cloud`.`volume_view` AS
        disk_offering.display_text disk_offering_display_text,
        disk_offering.use_local_storage,
        service_offering.system_use,
+        disk_offering.system_use,


Hi @DK101010 the build is failing because this column does not exist:

Caused by: java.sql.SQLSyntaxErrorException: Unknown column 'disk_offering.system_use' in 'field list' at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185) at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87) at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:215) ... 52 more 2022-04-07 13:39:12,242 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 369 Name = Upgrade; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-DatabaseUpgradeChecker.upgrade:314-DatabaseUpgradeChecker.check:389-CloudStackExtendedLifeCycle.checkIntegrity:64-CloudStackExtendedLifeCycle.start:54-DefaultLifecycleProcessor.doStart:178-DefaultLifecycleProcessor.access$200:54-DefaultLifecycleProcessor$LifecycleGroup.start:356-Iterable.forEach:75-DefaultLifecycleProcessor.startBeans:155

Hi @nvazquez, is deleted. Is cames not from me, I guess it was a little mistake during the merge.

@DK101010 ok, I see there are conflicts again, sorry will need further fixing :)

@nvazquez ok :D

…/add_vdisk_uuid_to_list_volume

github-actions · 2022-04-11T14:03:37Z

Hi @ravening, your pull request has merge conflicts. Can you fix the conflicts and sync your branch with the base branch?

…/add_vdisk_uuid_to_list_volume

nvazquez · 2022-04-12T10:13:50Z

@blueorangutan package

blueorangutan · 2022-04-12T10:26:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-04-12T11:06:55Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3142

weizhouapache

code lgtm

nvazquez · 2022-04-12T11:45:40Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2022-04-12T11:46:02Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2022-04-13T01:02:39Z

Trillian test result (tid-3870)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46598 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5848-t3870-vmware-67u3.zip
Smoke tests completed. 90 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_create_pvlan_network	`Error`	0.03	test_pvlan.py

nvazquez · 2022-04-13T15:27:58Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2022-04-13T15:29:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2022-04-14T00:46:55Z

Trillian test result (tid-3899)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32334 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5848-t3899-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py

nvazquez · 2022-04-14T04:40:03Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2022-04-14T04:42:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2022-04-14T14:40:34Z

Trillian test result (tid-3904)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34834 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5848-t3904-vmware-67u3.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2022-04-19T21:16:01Z

@nvazquez please review and merge this ?

sureshanaparti added this to the 4.17.0.0 milestone Jan 11, 2022

sureshanaparti added the component:vmware label Jan 11, 2022

apache deleted a comment from blueorangutan Jan 11, 2022

sureshanaparti requested review from nvazquez, DaanHoogland and harikrishna-patnala and removed request for nvazquez January 13, 2022 06:46

GutoVeronezi reviewed Jan 26, 2022

View reviewed changes

vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java Outdated Show resolved Hide resolved

nvazquez assigned DK101010 Feb 6, 2022

DK101010 added 2 commits February 11, 2022 07:40

get vdisk uuid from vcenter and store it into database

a841420

add vdisk uuid as external_uuid to listVolume response

743c5de

nvazquez reviewed Apr 7, 2022

View reviewed changes

DK101010 added 2 commits April 8, 2022 07:28

Merge branch 'main' of https://github.com/apache/cloudstack into feat…

ab0c9a7

…/add_vdisk_uuid_to_list_volume

remove line disk_offering.system_use

e4958e3

boring-cyborg bot added component:api component:storage labels Apr 8, 2022

github-actions bot added the status:has-conflicts label Apr 11, 2022

Merge branch 'main' of https://github.com/apache/cloudstack into feat…

866df1d

…/add_vdisk_uuid_to_list_volume

github-actions bot removed the status:has-conflicts label Apr 12, 2022

weizhouapache approved these changes Apr 12, 2022

View reviewed changes

DaanHoogland approved these changes Apr 13, 2022

View reviewed changes

nvazquez approved these changes Apr 20, 2022

View reviewed changes

nvazquez merged commit ccac1a3 into apache:main Apr 20, 2022

Feat/add vdisk UUID to list volume #5848

Feat/add vdisk UUID to list volume #5848

Conversation

DK101010 commented Jan 11, 2022 • edited by DaanHoogland Loading

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

sureshanaparti commented Jan 11, 2022

blueorangutan commented Jan 11, 2022

blueorangutan commented Jan 11, 2022

sureshanaparti commented Jan 11, 2022

blueorangutan commented Jan 11, 2022

blueorangutan commented Jan 11, 2022

DaanHoogland commented Jan 12, 2022

sureshanaparti commented Jan 13, 2022

blueorangutan commented Jan 13, 2022

blueorangutan commented Jan 13, 2022

nvazquez commented Feb 6, 2022

blueorangutan commented Feb 6, 2022

blueorangutan commented Feb 6, 2022

nvazquez commented Feb 7, 2022

DK101010 commented Feb 9, 2022

nvazquez commented Feb 9, 2022

DK101010 commented Feb 10, 2022

blueorangutan commented Apr 6, 2022

nvazquez commented Apr 7, 2022

blueorangutan commented Apr 7, 2022

blueorangutan commented Apr 7, 2022

nvazquez Apr 7, 2022

Choose a reason for hiding this comment

DK101010 Apr 8, 2022

Choose a reason for hiding this comment

nvazquez Apr 11, 2022

Choose a reason for hiding this comment

DK101010 Apr 12, 2022

Choose a reason for hiding this comment

github-actions bot commented Apr 11, 2022

nvazquez commented Apr 12, 2022

blueorangutan commented Apr 12, 2022

blueorangutan commented Apr 12, 2022

weizhouapache left a comment

Choose a reason for hiding this comment

nvazquez commented Apr 12, 2022

blueorangutan commented Apr 12, 2022

blueorangutan commented Apr 13, 2022

nvazquez commented Apr 13, 2022

blueorangutan commented Apr 13, 2022

blueorangutan commented Apr 14, 2022

nvazquez commented Apr 14, 2022

blueorangutan commented Apr 14, 2022

blueorangutan commented Apr 14, 2022

weizhouapache commented Apr 19, 2022

DK101010 commented Jan 11, 2022 •

edited by DaanHoogland

Loading