CLOUDSTACK-9842: Unable to map root volume usage to VM

[sudhansu7]

If a VM is cold migrated the vm_instance_id and uuid of volume is nullified. So there is no link between volume and vm.

With this the ROOT volume usage can not be mapped to a VM.

mysql>  select id,name,uuid,created,removed,state from cloud.volumes where name="ROOT-88" ;
+-----+------------+--------------------------------------+---------------------+---------------------+----------+
| id  | name       | uuid                                 | created             | instance_id         | state    |
+-----+------------+--------------------------------------+---------------------+---------------------+----------+
| 89  | ROOT-88   | NULL                                 | 2015-08-31 06:14:01 | NULL                | Expunged |
| 96  | ROOT-88   | a32e70e7-9869-49c8-92fa-5acd45abc94e | 2015-08-31 07:20:41 | 88                 | Expunged |
+-----+------------+--------------------------------------+---------------------+---------------------+----------+
2 rows in set (0.00 sec)

mysql> select * from cloud_usage.usage_volume where id in (89, 96);
+-----+---------+------------+-----------+------------------+-------------+-------------+---------------------+---------------------+
| id  | zone_id | account_id | domain_id | disk_offering_id | template_id | size        | created             | deleted             |
+-----+---------+------------+-----------+------------------+-------------+-------------+---------------------+---------------------+
| 89 |       1 |         55 |        54 |             NULL |        NULL | 16106127360 | 2015-08-31 06:14:01 | 2015-08-31 07:23:32 |
| 96 |       1 |         55 |        54 |               16 |        NULL | 16106127360 | 2015-08-31 07:23:32 | 2015-08-31 08:09:40 |
+-----+---------+------------+-----------+------------------+-------------+-------------+---------------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from cloud_usage.cloud_usage where usage_id in (89, 96)\G;
*************************** 1. row ***************************
            id: 48042
       zone_id: 1
    account_id: 55
     domain_id: 54
   description: Volume Id: 96 usage time (DiskOffering: 16)
 usage_display: 0.768889 Hrs
    usage_type: 6
     raw_usage: 0.768889188766479
vm_instance_id: NULL
       vm_name: NULL
   offering_id: 16
   template_id: NULL
      usage_id: 96
          type: NULL
          size: 16106127360
    network_id: NULL
    start_date: 2015-08-30 15:00:00
      end_date: 2015-08-31 14:59:59
  virtual_size: NULL
*************************** 2. row ***************************
            id: 48044
       zone_id: 1
    account_id: 55
     domain_id: 54
   description: Volume Id: 89 usage time
 usage_display: 1.158611 Hrs
    usage_type: 6
     raw_usage: 1.15861141681671
vm_instance_id: NULL
       vm_name: NULL
   offering_id: NULL
   template_id: NULL
      usage_id: 89
          type: NULL
          size: 16106127360
    network_id: NULL
    start_date: 2015-08-30 15:00:00
      end_date: 2015-08-31 14:59:59
  virtual_size: NULL
2 rows in set (0.06 sec)

Usage id 48044 is for volume id  89. Volume 89 does not have any instance_id. 

Solution:
Removed code which nullifies vm_instance_id
Also modified QueryManagerImpl to ignore volume which does not have uuid. This is to avoid duplicate volume listing.

[SudharmaJain]

LGTM for the code changes.

[borisstoyanov]

@blueorangutan package

[blueorangutan]

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-609

[borisstoyanov]

@blueorangutan test

[blueorangutan]

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-972)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34338 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2012-t972-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 47 look ok, 2 have error(s)

Test	Result	Time (s)	Test File
test_02_redundant_VPC_default_routes	Failure	862.85	test_vpc_redundant.py
test_04_rvpc_privategw_static_routes	Failure	349.65	test_privategw_acl.py
test_01_vpc_site2site_vpn	Success	169.65	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	Success	65.81	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	Success	250.32	test_vpc_vpn.py
test_02_VPC_default_routes	Success	290.91	test_vpc_router_nics.py
test_01_VPC_nics_after_destroy	Success	521.48	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	Success	516.65	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Success	1410.10	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Success	548.05	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Success	1289.05	test_vpc_redundant.py
test_09_delete_detached_volume	Success	156.11	test_volumes.py
test_08_resize_volume	Success	156.09	test_volumes.py
test_07_resize_fail	Success	161.09	test_volumes.py
test_06_download_detached_volume	Success	155.99	test_volumes.py
test_05_detach_volume	Success	155.62	test_volumes.py
test_04_delete_attached_volume	Success	150.93	test_volumes.py
test_03_download_attached_volume	Success	155.98	test_volumes.py
test_02_attach_volume	Success	94.81	test_volumes.py
test_01_create_volume	Success	710.91	test_volumes.py
test_03_delete_vm_snapshots	Success	275.18	test_vm_snapshots.py
test_02_revert_vm_snapshots	Success	95.72	test_vm_snapshots.py
test_01_create_vm_snapshots	Success	158.72	test_vm_snapshots.py
test_deploy_vm_multiple	Success	257.11	test_vm_life_cycle.py
test_deploy_vm	Success	0.02	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.02	test_vm_life_cycle.py
test_10_attachAndDetach_iso	Success	26.54	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.17	test_vm_life_cycle.py
test_08_migrate_vm	Success	45.69	test_vm_life_cycle.py
test_07_restore_vm	Success	0.09	test_vm_life_cycle.py
test_06_destroy_vm	Success	125.63	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.65	test_vm_life_cycle.py
test_02_start_vm	Success	10.13	test_vm_life_cycle.py
test_01_stop_vm	Success	40.25	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	45.36	test_templates.py
test_08_list_system_templates	Success	0.03	test_templates.py
test_07_list_public_templates	Success	0.03	test_templates.py
test_05_template_permissions	Success	0.04	test_templates.py
test_04_extract_template	Success	5.14	test_templates.py
test_03_delete_template	Success	5.08	test_templates.py
test_02_edit_template	Success	90.16	test_templates.py
test_01_create_template	Success	40.32	test_templates.py
test_10_destroy_cpvm	Success	161.56	test_ssvm.py
test_09_destroy_ssvm	Success	163.52	test_ssvm.py
test_08_reboot_cpvm	Success	131.56	test_ssvm.py
test_07_reboot_ssvm	Success	133.49	test_ssvm.py
test_06_stop_cpvm	Success	131.62	test_ssvm.py
test_05_stop_ssvm	Success	133.99	test_ssvm.py
test_04_cpvm_internals	Success	1.25	test_ssvm.py
test_03_ssvm_internals	Success	3.49	test_ssvm.py
test_02_list_cpvm_vm	Success	0.09	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.09	test_ssvm.py
test_02_list_snapshots_with_removed_data_store	Success	86.56	test_snapshots.py
test_01_snapshot_root_disk	Success	11.02	test_snapshots.py
test_04_change_offering_small	Success	210.46	test_service_offerings.py
test_03_delete_service_offering	Success	0.03	test_service_offerings.py
test_02_edit_service_offering	Success	0.04	test_service_offerings.py
test_01_create_service_offering	Success	0.08	test_service_offerings.py
test_02_sys_template_ready	Success	0.09	test_secondary_storage.py
test_01_sys_vm_start	Success	0.12	test_secondary_storage.py
test_09_reboot_router	Success	35.23	test_routers.py
test_08_start_router	Success	30.21	test_routers.py
test_07_stop_router	Success	10.12	test_routers.py
test_06_router_advanced	Success	0.04	test_routers.py
test_05_router_basic	Success	0.03	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.69	test_routers.py
test_03_restart_network_cleanup	Success	60.39	test_routers.py
test_02_router_internal_adv	Success	1.14	test_routers.py
test_01_router_internal_basic	Success	0.58	test_routers.py
test_router_dns_guestipquery	Success	76.69	test_router_dns.py
test_router_dns_externalipquery	Success	0.06	test_router_dns.py
test_router_dhcphosts	Success	276.40	test_router_dhcphosts.py
test_router_dhcp_opts	Success	21.56	test_router_dhcphosts.py
test_01_updatevolumedetail	Success	0.07	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	130.71	test_reset_vm_on_reboot.py
test_createRegion	Success	0.03	test_regions.py
test_create_pvlan_network	Success	5.15	test_pvlan.py
test_dedicatePublicIpRange	Success	0.29	test_public_ip_range.py
test_03_vpc_privategw_restart_vpc_cleanup	Success	483.13	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Success	368.57	test_privategw_acl.py
test_01_vpc_privategw_acl	Success	81.49	test_privategw_acl.py
test_03_migration_options_storage_tags	Success	161.35	test_primary_storage.py
test_02_edit_primary_storage_tags	Success	0.07	test_primary_storage.py
test_01_primary_storage_nfs	Success	35.73	test_primary_storage.py
test_01_deploy_vms_storage_tags	Success	30.73	test_primary_storage.py
test_createPortablePublicIPRange	Success	15.13	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.36	test_portable_publicip.py
test_isolate_network_password_server	Success	91.17	test_password_server.py
test_UpdateStorageOverProvisioningFactor	Success	0.22	test_over_provisioning.py
test_oobm_zchange_password	Success	30.48	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Success	16.27	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	10.18	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	15.24	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	15.24	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	15.23	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.30	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	15.41	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	87.16	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.12	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	0.07	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	5.14	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.08	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.06	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.06	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	23.30	test_outofbandmanagement.py
test_extendPhysicalNetworkVlan	Success	15.23	test_non_contigiousvlan.py
test_01_nic	Success	438.11	test_nic.py
test_releaseIP	Success	262.11	test_network.py
test_reboot_router	Success	417.80	test_network.py
test_public_ip_user_account	Success	10.20	test_network.py
test_public_ip_admin_account	Success	40.22	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	Success	66.71	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	Success	61.86	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	Success	123.84	test_network.py
test_delete_account	Success	292.13	test_network.py
test_02_port_fwd_on_non_src_nat	Success	55.48	test_network.py
test_01_port_fwd_on_src_nat	Success	111.71	test_network.py
test_nic_secondaryip_add_remove	Success	212.35	test_multipleips_per_nic.py
login_test_saml_user	Success	17.79	test_login.py
test_assign_and_removal_lb	Success	133.54	test_loadbalance.py
test_02_create_lb_rule_non_nat	Success	187.24	test_loadbalance.py
test_01_create_lb_rule_src_nat	Success	227.96	test_loadbalance.py
test_03_list_snapshots	Success	0.04	test_list_ids_parameter.py
test_02_list_templates	Success	0.03	test_list_ids_parameter.py
test_01_list_volumes	Success	0.02	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.04	test_iso.py
test_05_iso_permissions	Success	0.04	test_iso.py
test_04_extract_Iso	Success	5.11	test_iso.py
test_03_delete_iso	Success	95.13	test_iso.py
test_02_edit_iso	Success	0.04	test_iso.py
test_01_create_iso	Success	20.68	test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Success	192.61	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Success	142.53	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Success	494.59	test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	Success	413.86	test_internal_lb.py
test_dedicateGuestVlanRange	Success	10.20	test_guest_vlan_range.py
test_UpdateConfigParamWithScope	Success	0.10	test_global_settings.py
test_rolepermission_lifecycle_update	Success	5.80	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	5.69	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	5.57	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	5.61	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	5.66	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	5.77	test_dynamicroles.py
test_role_lifecycle_update	Success	10.66	test_dynamicroles.py
test_role_lifecycle_list	Success	5.61	test_dynamicroles.py
test_role_lifecycle_delete	Success	10.63	test_dynamicroles.py
test_role_lifecycle_create	Success	5.60	test_dynamicroles.py
test_role_inuse_deletion	Success	5.66	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	6.91	test_dynamicroles.py
test_role_account_acls	Success	7.03	test_dynamicroles.py
test_default_role_deletion	Success	5.65	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.05	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.03	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.04	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.05	test_disk_offerings.py
test_01_create_disk_offering	Success	0.08	test_disk_offerings.py
test_deployvm_userdispersing	Success	20.39	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	35.47	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	55.45	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	10.28	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	65.58	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	5.66	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	5.65	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	216.78	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	216.80	test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup	Success	65.81	test_affinity_groups.py
test_change_service_offering_for_vm_with_snapshots	Skipped	0.00	test_vm_snapshots.py
test_01_test_vm_volume_snapshot	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_11_ss_nfs_version_on_ssvm	Skipped	0.02	test_ssvm.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.03	test_primary_storage.py
test_nested_virtualization_vmware	Skipped	0.00	test_nested_virtualization.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.02	test_deploy_vgpu_enabled_vm.py
test_3d_gpu_support	Skipped	0.02	test_deploy_vgpu_enabled_vm.py

[borisstoyanov]

LGMT based on code review and test results

[yvsubhash]

Tested this change. LGTM for test

[sudhansu7]

tag:mergeready

[cloudmonger]

ACS CI BVT Run

Sumarry:
Build Number 784
Hypervisor xenserver
NetworkType Advanced
Passed=108
Failed=2
Skipped=12

Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

• test_list_ids_parameter.py

• ContextSuite context=TestListIdsParams>:setup Failing since 23 runs

• test_volumes.py

• test_06_download_detached_volume Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_routers_network_ops.py
test_disk_offerings.py