Prevent starting a VM in destroyed state (or any state but Stopped) #5165

Pearl1594 · 2021-06-29T13:00:50Z

Description

A destroyed VM when attempted to be started (using the bulk action support in the UI), results in a successful operation, however, the VM doesn't actually start. This PR fixes the false positive result shown in the above mentioned condition

VM continues to remain in destroyed state - but a success response is returned

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tried starting an instance in destroyed state:

2021-06-29 12:57:51,609 WARN  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-da4601fa job-52/job-54 ctx-cc08e96e) (logid:08e57fcb) VM VM instance {"id": "3", "name": "i-2-3-VM", "uuid": "4213c3d9-72a8-4201-96c4-003e2c2f845b", "type"="User"} is not in a state to be started: Destroyed
2021-06-29 12:57:51,610 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-da4601fa job-52/job-54 ctx-cc08e96e) (logid:08e57fcb) Caught CloudRuntimeException, returning job failed com.cloud.utils.exception.CloudRuntimeException: Cannot start VM: VM instance {"id": "3", "name": "i-2-3-VM", "uuid": "4213c3d9-72a8-4201-96c4-003e2c2f845b", "type"="User"} in Destroyed state
2021-06-29 12:57:51,613 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-5:ctx-da4601fa job-52/job-54 ctx-cc08e96e) (logid:08e57fcb) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"rawParams":{"VmPassword":"rO0ABXQADnNhdmVkX3Bhc3N3b3Jk"},"userId":2,"accountId":2,"vmId":3,"handlerName":"VirtualMachineManagerImpl"}

Pearl1594 · 2021-06-29T13:02:30Z

@blueorangutan package

blueorangutan · 2021-06-29T13:03:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-06-29T13:46:06Z

Packaging result: ✔️ centos7 ✖️ centos8 ✔️ debian. SL-JID 422

Pearl1594 · 2021-06-29T13:47:30Z

@blueorangutan package

blueorangutan · 2021-06-29T13:48:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-06-29T14:13:05Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 424

Pearl1594 · 2021-06-29T14:15:00Z

@blueorangutan test

blueorangutan · 2021-06-29T14:16:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Pearl1594 · 2021-06-29T15:01:24Z

@blueorangutan package

blueorangutan · 2021-06-29T15:02:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-06-29T15:28:51Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 425

blueorangutan · 2021-06-29T16:43:08Z

Trillian Build Failed (tid-1159)

Pearl1594 · 2021-06-29T16:46:28Z

@blueorangutan test

blueorangutan · 2021-06-29T16:47:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

GutoVeronezi · 2021-06-29T19:36:46Z

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

+                s_logger.warn("VM " + vm + " is not in a state to be started: " + state);
+                throw new CloudRuntimeException(String.format("Cannot start VM: %s in %s state", vm, state));


We could unify these messages in a String var (with String.format).

Suggested change

s_logger.warn("VM " + vm + " is not in a state to be started: " + state);

throw new CloudRuntimeException(String.format("Cannot start VM: %s in %s state", vm, state));

String msg = String.format("Cannot start VM: %s in %s state", vm, state)

s_logger.warn(msg);

throw new CloudRuntimeException(msg));

blueorangutan · 2021-06-30T07:43:27Z

Trillian test result (tid-1160)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 51927 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5165-t1160-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 84 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	224.01	test_kubernetes_clusters.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	347.60	test_routers_network_ops.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	537.61	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	537.63	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	474.42	test_vpc_redundant.py
test_01_vpc_site2site_vpn_multiple_options	`Failure`	380.14	test_vpc_vpn.py

Pearl1594 · 2021-07-01T04:05:11Z

@blueorangutan package

blueorangutan · 2021-07-01T04:06:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-07-01T04:30:33Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 438

Pearl1594 · 2021-07-01T04:31:13Z

@blueorangutan test

blueorangutan · 2021-07-01T04:32:02Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-01T15:21:33Z

Trillian test result (tid-1179)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37293 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5165-t1179-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_nested_virtualization.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_portforwardingrules.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Smoke tests completed. 72 look OK, 16 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestLoadBalance>:setup	`Error`	0.00	test_loadbalance.py
test_list_vms_metrics	`Error`	0.24	test_metrics_api.py
ContextSuite context=TestNetworkACL>:setup	`Error`	0.00	test_network_acl.py
test_delete_account	`Error`	0.46	test_network.py
test_delete_network_while_vm_on_it	`Error`	1.12	test_network.py
test_deploy_vm_l2network	`Error`	1.11	test_network.py
test_l2network_restart	`Error`	2.19	test_network.py
ContextSuite context=TestPortForwarding>:setup	`Error`	3.41	test_network.py
ContextSuite context=TestPublicIP>:setup	`Error`	1.25	test_network.py
test_reboot_router	`Failure`	0.04	test_network.py
test_releaseIP	`Error`	0.44	test_network.py
ContextSuite context=TestRouterRules>:setup	`Error`	0.49	test_network.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_06_deploy_invalid_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	`Error`	1802.89	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
test_nic_secondaryip_add_remove	`Failure`	0.04	test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup	`Error`	0.00	test_nested_virtualization.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup	`Error`	0.00	test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup	`Error`	0.00	test_portforwardingrules.py
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
test_09_project_suspend	`Error`	1.08	test_projects.py
test_10_project_activation	`Error`	1.06	test_projects.py
ContextSuite context=TestResetVmOnReboot>:setup	`Error`	0.00	test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup	`Error`	0.00	test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup	`Error`	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	`Error`	0.00	test_router_dhcphosts.py

Pearl1594 · 2021-07-01T15:42:50Z

@blueorangutan test

blueorangutan · 2021-07-01T15:43:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-02T04:14:33Z

Trillian test result (tid-1190)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43413 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5165-t1190-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 86 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	`Failure`	336.32	test_routers_network_ops.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	523.40	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	507.98	test_vpc_redundant.py

DaanHoogland

cltgm

GutoVeronezi

CLGTM; I raised a last point about toString.

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

Pearl1594 · 2021-07-05T13:46:51Z

@blueorangutan package

blueorangutan · 2021-07-05T13:47:02Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-07-05T14:09:06Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 475

Pearl1594 · 2021-07-05T15:50:18Z

@blueorangutan test

blueorangutan · 2021-07-05T15:51:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-06T01:31:01Z

Trillian test result (tid-1205)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33120 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5165-t1205-kvm-centos7.zip
Smoke tests completed. 88 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

sureshanaparti · 2021-07-06T06:16:09Z

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

-                return null;
+                String msg = String.format("Cannot start %s in %s state", vm, state);
+                s_logger.warn(msg);
+                throw new CloudRuntimeException(msg);


vladimirpetrov

Functionality tested - LGTM.

GabrielBrascher

Code LGTM

sureshanaparti · 2021-07-09T09:39:53Z

Merging this based on LGTMs, manual and smoke tests.

Pearl1594 requested review from vladimirpetrov and DaanHoogland June 29, 2021 13:00

Prevent starting a VM in destroyed state (or any state but Stopped)

bc9539c

Pearl1594 force-pushed the fix-startVm branch from 484e97a to bc9539c Compare June 29, 2021 14:57

GutoVeronezi reviewed Jun 29, 2021

View reviewed changes

unify error mgs and exception

220d8bc

DaanHoogland approved these changes Jul 2, 2021

View reviewed changes

rohityadavcloud added this to the 4.16.0.0 milestone Jul 5, 2021

GutoVeronezi approved these changes Jul 5, 2021

View reviewed changes

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java Outdated Show resolved Hide resolved

fix

826f36b

sureshanaparti reviewed Jul 6, 2021

View reviewed changes

sureshanaparti approved these changes Jul 6, 2021

View reviewed changes

vladimirpetrov approved these changes Jul 7, 2021

View reviewed changes

GabrielBrascher approved these changes Jul 7, 2021

View reviewed changes

GabrielBrascher added the status:ready-for-merge label Jul 7, 2021

GabrielBrascher assigned Pearl1594 Jul 7, 2021

sureshanaparti merged commit 3fd9250 into apache:main Jul 9, 2021

		s_logger.warn("VM " + vm + " is not in a state to be started: " + state);
		throw new CloudRuntimeException(String.format("Cannot start VM: %s in %s state", vm, state));

Prevent starting a VM in destroyed state (or any state but Stopped) #5165

Prevent starting a VM in destroyed state (or any state but Stopped) #5165

Conversation

Pearl1594 commented Jun 29, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Pearl1594 commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

Pearl1594 commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

Pearl1594 commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

Pearl1594 commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

Pearl1594 commented Jun 29, 2021

blueorangutan commented Jun 29, 2021

GutoVeronezi Jun 29, 2021

Choose a reason for hiding this comment

DaanHoogland Jun 30, 2021

Choose a reason for hiding this comment

blueorangutan commented Jun 30, 2021

Pearl1594 commented Jul 1, 2021

blueorangutan commented Jul 1, 2021

blueorangutan commented Jul 1, 2021

Pearl1594 commented Jul 1, 2021

blueorangutan commented Jul 1, 2021

blueorangutan commented Jul 1, 2021

Pearl1594 commented Jul 1, 2021

blueorangutan commented Jul 1, 2021

blueorangutan commented Jul 2, 2021

DaanHoogland left a comment

Choose a reason for hiding this comment

GutoVeronezi left a comment

Choose a reason for hiding this comment

Pearl1594 commented Jul 5, 2021

blueorangutan commented Jul 5, 2021

blueorangutan commented Jul 5, 2021

Pearl1594 commented Jul 5, 2021

blueorangutan commented Jul 5, 2021

blueorangutan commented Jul 6, 2021

sureshanaparti Jul 6, 2021

Choose a reason for hiding this comment

vladimirpetrov left a comment

Choose a reason for hiding this comment

GabrielBrascher left a comment

Choose a reason for hiding this comment

sureshanaparti commented Jul 9, 2021