Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service #6234

Pearl1594 · 2022-04-08T09:32:44Z

Description

This PR fixes an issue observed during expunging a VM deployed on a network using COnfigDrive userdata service and host cache

2022-04-08 07:24:14,834 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-8:ctx-e4d333b0 job-78) (logid:a6b8b1aa) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Unable to get an answer to handle config drive deletion for vm: i-2-8-VM on host: 1
        at com.cloud.network.element.ConfigDriveNetworkElement.deleteConfigDriveIsoOnHostCache(ConfigDriveNetworkElement.java:569)
        at com.cloud.network.element.ConfigDriveNetworkElement.deleteConfigDriveIso(ConfigDriveNetworkElement.java:638)
        at com.cloud.network.element.ConfigDriveNetworkElement.postStateTransitionEvent(ConfigDriveNetworkElement.java:321)
        at com.cloud.network.element.ConfigDriveNetworkElement.postStateTransitionEvent(ConfigDriveNetworkElement.java:89)
        at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:142)
        at com.cloud.vm.VirtualMachineManagerImpl.stateTransitTo(VirtualMachineManagerImpl.java:2129)
        at com.cloud.vm.VirtualMachineManagerImpl$5.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:2160)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.vm.VirtualMachineManagerImpl.destroy(VirtualMachineManagerImpl.java:2150)

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Post Fix:
Successfully expunges the VM

022-04-08 08:22:29,583 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-2:ctx-d3f0bdc8 job-94 ctx-6bcc9fd2) (logid:4d070eb7) Asking ConfigDrive to release NicProfile {"broadcastUri":null,"iPv4Address":"10.1.1.226","id":18,"reservationId":"9b5d5de0-a880-4ee4-b906-6fa086b19010","vmId":8}
2022-04-08 08:22:29,584 DEBUG [c.c.n.e.ConfigDriveNetworkElement] (API-Job-Executor-2:ctx-d3f0bdc8 job-94 ctx-6bcc9fd2) (logid:4d070eb7) Deleting config drive ISO for vm: i-2-8-VM on host: 1
2022-04-08 08:22:29,587 WARN  [c.c.n.e.ConfigDriveNetworkElement] (API-Job-Executor-2:ctx-d3f0bdc8 job-94 ctx-6bcc9fd2) (logid:4d070eb7) Host 1 appears to be unavailable, skipping deletion of config-drive ISO on host cache
2022-04-08 08:22:29,595 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-d3f0bdc8 job-94 ctx-6bcc9fd2) (logid:4d070eb7) Sync job-96 execution on object VmWorkJobQueue.8
2022-04-08 08:22:31,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-dd836c4c) (logid:00038e1f) Execute sync-queue item: SyncQueueItemVO {id:34, queueId: 9, contentType: AsyncJob, contentId: 96, lastProcessMsid: 32986976224091, lastprocessNumber: 10, lastProcessTime: Fri Apr 08 08:22:31 UTC 2022, created: Fri Apr 08 08:22:29 UTC 2022}
2022-04-08 08:22:31,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-dd836c4c) (logid:00038e1f) Schedule queued job-96
2022-04-08 08:22:31,508 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-a7f3e64f job-94/job-96) (logid:030ae26c) Add job-96 into job monitoring
2022-04-08 08:22:31,511 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-a7f3e64f job-94/job-96) (logid:4d070eb7) Executing AsyncJobVO: {id:96, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAACHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32986976224091, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Apr 08 08:22:29 UTC 2022, removed: null}

…igDrive userdata service

Pearl1594 · 2022-04-08T09:42:59Z

@blueorangutan package

blueorangutan · 2022-04-08T09:45:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-04-08T10:30:59Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3101

Pearl1594 · 2022-04-08T10:31:40Z

@blueorangutan test

blueorangutan · 2022-04-08T10:32:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-04-08T19:42:31Z

Trillian test result (tid-3845)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31590 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6234-t3845-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez

@Pearl1594 I'm getting this error while expunging a VM on userdata network:

2022-04-11 14:00:22,740 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-5b70a0d9) (logid:9468353d) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$Behin
dOnPingListener
2022-04-11 14:00:22,740 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-5b70a0d9) (logid:9468353d) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$SetHo
stParamsListener
2022-04-11 14:00:22,740 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-5b70a0d9) (logid:9468353d) Sending Disconnect to listener: com.cloud.hypervisor.kvm.discoverer.LxcServerD
iscoverer
2022-04-11 14:00:22,740 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-5b70a0d9) (logid:9468353d) Sending Disconnect to listener: com.cloud.hypervisor.kvm.discoverer.KvmServerD
iscoverer
2022-04-11 14:00:22,740 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-5b70a0d9) (logid:9468353d) Transition:[Resource state = Enabled, Agent event = ShutdownRequested, Host id
 = 4, name = s-2-VM]
2022-04-11 14:00:22,740 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-29:ctx-87402a4c job-39) (logid:e948e5a5) Unexpected exception while executing org.apache.cloudstack.api.c
ommand.admin.vm.DestroyVMCmdByAdmin
java.lang.NullPointerException
        at com.cloud.network.element.ConfigDriveNetworkElement.deleteConfigDriveIso(ConfigDriveNetworkElement.java:674)
        at com.cloud.network.element.ConfigDriveNetworkElement.postStateTransitionEvent(ConfigDriveNetworkElement.java:322)
        at com.cloud.network.element.ConfigDriveNetworkElement.postStateTransitionEvent(ConfigDriveNetworkElement.java:90)
        at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:142)
        at com.cloud.vm.VirtualMachineManagerImpl.stateTransitTo(VirtualMachineManagerImpl.java:2183)
        at com.cloud.vm.VirtualMachineManagerImpl$5.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:2214)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.vm.VirtualMachineManagerImpl.destroy(VirtualMachineManagerImpl.java:2204)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.destroyVirtualMachine(VMEntityManagerImpl.java:283)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.destroy(VirtualMachineEntityImpl.java:235)
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:5356)
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3214)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

nvazquez

LGTM - ignore last comment, had an issue with SSVM, retried and worked as expected

…igDrive userdata service (apache#6234) (cherry picked from commit b6072fc) Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Allow expunging a VM on a deleted host when using host cache and Conf…

524fec5

…igDrive userdata service

boring-cyborg bot added the component:networking label Apr 8, 2022

Pearl1594 requested a review from nvazquez April 8, 2022 09:33

yadvr approved these changes Apr 8, 2022

View reviewed changes

nvazquez added this to the 4.17.0.0 milestone Apr 11, 2022

nvazquez added the type:bug label Apr 11, 2022

nvazquez reviewed Apr 11, 2022

View reviewed changes

nvazquez approved these changes Apr 11, 2022

View reviewed changes

nvazquez merged commit b6072fc into apache:4.16 Apr 11, 2022

yadvr mentioned this pull request May 29, 2023

Destroy vm API errors with failure to delete ConfigDrive #7428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service #6234

Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service #6234

Uh oh!

Pearl1594 commented Apr 8, 2022 •

edited

Loading

Uh oh!

Pearl1594 commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

Pearl1594 commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

nvazquez left a comment

Uh oh!

nvazquez left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service #6234

Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service #6234

Uh oh!

Conversation

Pearl1594 commented Apr 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Pearl1594 commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

Pearl1594 commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

blueorangutan commented Apr 8, 2022

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pearl1594 commented Apr 8, 2022 •

edited

Loading