Deploying a VM in dedicated domain ends in error

[tdtmusic2]

Hi all,

I’m trying to deploy vms on a different domain than / and something is not right, deployments end in errors. The setup is simple, one zone dedicated to a user (domain admin), one pod and cluster, 2 hosts (all dedicated to that domain). Storage type is local. When I try to deploy a vm and look in the logs I get this:

2025-06-24 10:33:21,351 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Setting VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}] password to a randomly generated password.
2025-06-24 10:33:21,354 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Trying to deploy VM [error decoding VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}] and details: Plan [{"_dcId":7,"_recreateDisks":false,"preferredHostIds":[],"migrationPlan":false,"hostPriorities":{}}]; avoid list [{}] and planner: [].
2025-06-24 10:33:21,355 DEBUG [c.c.a.ApiServlet] (qtp1513608173-7488:[ctx-4ab88968, ctx-1e407ccc]) (logid:56e217ae) ===END===  86.104.176.186 -- GET  jobId=897c87bd-aad9-4a6f-950b-5ee47fef93f9&command=queryAsyncJobResult&response=json&sessionkey=u7m6RhfS1GqB5Jweqh9wACKTz6w
2025-06-24 10:33:21,355 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Checking non dedicated resources to deploy VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}].
2025-06-24 10:33:21,356 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Zone [9e0eae3b-acd3-498e-8300-518c6905c3de] is dedicated. Checking if account [15] in domain [6] can use this zone to deploy VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}].
2025-06-24 10:33:21,359 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Adding pods [[7]], clusters [[7]] and hosts [[22,26]] to the avoid list in the deploy process of user VM [error decoding VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}], because this VM is not explicitly dedicated to these components.
2025-06-24 10:33:21,360 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Trying to allocate a host and storage pools from datacenter [Zone {"id": "7", "name": "UPFR", "uuid": "b149eae3-d439-411f-ba2b-63187f6e10a9"}], pod [null], cluster [null], to deploy VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}] with requested CPU [10756] and requested RAM [(8.00 GB) 8589934592].
2025-06-24 10:33:21,360 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) ROOT volume [Volume {"id":178,"instanceId":190,"name":"ROOT-190","uuid":"3bf2ff53-50bb-4677-b246-b2d932ef1ca7","volumeType":"ROOT"}] is not ready to deploy VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}].
2025-06-24 10:33:21,362 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Adding pods [] to the avoid set because these pods are in the Disabled state.
2025-06-24 10:33:21,363 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Adding clusters [] of pod [7] to the void set because these clusters are in the Disabled state.
2025-06-24 10:33:21,363 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Adding hosts [] of datacenter [b149eae3-d439-411f-ba2b-63187f6e10a9] to the avoid set, because these hosts are in the Disabled state.
2025-06-24 10:33:21,364 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) DeploymentPlan [DataCenterDeployment] has not specified host. Trying to find another destination to deploy VM [VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}], avoiding pods [7], clusters [7] and hosts [22, 26].
2025-06-24 10:33:21,365 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Deploy avoids pods: [7], clusters: [7], hosts: [22, 26].
2025-06-24 10:33:21,365 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Deploy hosts with priorities {}, hosts have NORMAL priority by default
2025-06-24 10:33:21,365 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Searching all possible resources under this Zone: Zone {"id": "7", "name": "UPFR", "uuid": "b149eae3-d439-411f-ba2b-63187f6e10a9"}
2025-06-24 10:33:21,365 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 7
2025-06-24 10:33:21,368 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) No clusters found having a host with enough capacity, returning.
2025-06-24 10:33:21,369 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Destroying vm VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"} as it failed to create on Host: null with id null
2025-06-24 10:33:21,382 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Error","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"} state transited from [Stopped] to [Error] with event [OperationFailedToError]. VM's original host: null, new host: null, host before state transition: null
2025-06-24 10:33:21,383 DEBUG [c.c.s.d.VolumeDaoImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Removing volume 178 from DB
2025-06-24 10:33:21,387 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Updating resource Type = volume count for Account = 15 Operation = decreasing Amount = 1
2025-06-24 10:33:21,390 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Updating resource Type = primary_storage count for Account = 15 Operation = decreasing Amount = (100.00 GB) 107374182400
2025-06-24 10:33:21,397 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) alertType=[8] dataCenterId=[7] podId=[null] clusterId=[null] message=[Failed to deploy Vm VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Error","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}, on Host null with Id: null].
2025-06-24 10:33:21,399 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Failed to deploy Vm VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Error","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}, on Host null with Id: null] and content [Failed to deploy Vm VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Error","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}, on Host null with Id: null].
2025-06-24 10:33:21,400 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Updating resource Type = user_vm count for Account = 15 Operation = decreasing Amount = 1
2025-06-24 10:33:21,403 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Updating resource Type = cpu count for Account = 15 Operation = decreasing Amount = 4
2025-06-24 10:33:21,406 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Updating resource Type = memory count for Account = 15 Operation = decreasing Amount = 8192
2025-06-24 10:33:21,410 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Skipping delivering event Event {"description":"{\"details\":\"Vm Id: d97f71b9-9e8b-41cb-911f-66f872c6fcec\",\"event\":\"VM.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"VM.CREATE","eventUuid":null,"resourceType":"VirtualMachine","resourceUUID":null} to any webhook as account ID is missing
2025-06-24 10:33:21,410 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) Failed to publish event [category: ActionEvent, type: VM.CREATE] on bus webhookEventBus
2025-06-24 10:33:21,414 INFO  [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) No destination found for a deployment for VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}: No destination found for a deployment for VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}
2025-06-24 10:33:21,414 DEBUG [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155, ctx-358c8143]) (logid:897c87bd) No destination found for a deployment for VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"} com.cloud.exception.InsufficientServerCapacityException: No destination found for a deployment for VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}Scope=interface com.cloud.dc.DataCenter; id=7
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:227)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5637)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5490)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5091)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5080)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy210.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:781)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-06-24 10:33:21,415 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-61:[ctx-50d9a6b0, job-3155]) (logid:897c87bd) Complete async job-3155, jobStatus: FAILED, resultCode: 533, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"533","errortext":"No destination found for a deployment for VM instance {"id":190,"instanceName":"i-15-190-VM","state":"Stopped","type":"User","uuid":"d97f71b9-9e8b-41cb-911f-66f872c6fcec"}"}

I think it might have something to do with the offering but I'm not sure. What am I doing wrong here?

Thank you!

[tdtmusic2]

Some more info here. If both the 2 hosts and their primary storages have tags (to control where vms are deployed), deployment fails no matter what I do, even if the tags are correct on the offering. If I remove the host tag and just leave the storage tag on the offering, deployment works on 1st host. On the second host it fails, however I think this is a different matter, I saw this error somewhere in the log: Unable to add bridge br2 port vnet13: Operation not supported","wait":"0","bypassHostMaintenance":"false"}}] }
What I don't understand is why it doesn't work with host tags, seems a good way to filter the destinations.

[DaanHoogland]

@tdtmusic2 , if host tags and a dedicated zone are a conflict that sounds like a bug, but it might be a design limitation as well. I see no error that explains why no destination is found in the log. Can you share the offering?

[tdtmusic2]

Local storage will always be specific to a single host. It would have to select the localhost for the host it deploys on at all times. never the other host. This should not be different when deploying from an ISO. The root disk is still selected the same way.

Also , how is this specific to dedication of a domain? In the original message you mentioned this behaviour is in a dedicated zone. Is this not like this in a “normal” zone?

I am not mixing hosts with local storages, but since each host has 2 storages I want the user to be able to choose where to deploy, on ssd storage or hdd storage, on each host. That's why I am using tags.
The domain is a new domain created under root and the zone is dedicated to that domain. There is a user with domain admin role for that domain. THat line in the logs says "Checking non dedicated resources to deploy VM ", that's why I thought it also has something to do with the fact that the zone is dedicated.

[DaanHoogland]

ok, when a deployment in a dedicated resource fails in general other resources will be queried to see if deployment is possible there. So the tagging problem is not specifc to the dedication (I think).

[tdtmusic2]

ok, when a deployment in a dedicated resource fails in general other resources will be queried to see if deployment is possible there. So the tagging problem is not specifc to the dedication (I think).

Ok, so then what would be the proper way to solve this issue? At the moment I found a working setup like this: both ssd storages, from both hosts, have the same tag, ssd, and both hdd storages have hdd. Offerings are created with storage tag. If I deploy a vm AND choose the host manually, it uses the correct storage, depending on the offering that I choose. This is ok, but I cannot find a way to have ACS choose the host automatically - if I use host tags AND storage tags, even if they are correctly assigned, all deployments fail.

[DaanHoogland]

this as well sounds like a bug. you should be able to force selection by tags, whether in dedicated or in generic zones.

[DaanHoogland]

I created an issue, but it need reproducing to be sure about what is the underlying bug. see #11083