Cloudstack Agent Alert state

[muthukrishnang1100]

I have a total of three CloudStack agent hosts. All of them are repeatedly going into the Alert state.
When I perform the recovery steps, the hosts return to the Up state, but after 10 to 15 minutes they go back to Alert again.
This issue is happening on all agent hosts and is continuous.

[DaanHoogland]

@muthukrishnang1100 , any reason they go in alert state? any alert messages? any logs to share?

[muthukrishnang1100]

After reboot, Now hosts are up state. But, before we deleted the system vms, now we again start the system vms, it is still starting state more than a 1 day. no error showing in both management and agent logs, how we find the issue?

[muthukrishnang1100]

While starting the system VMs these error are commin.

2026-02-04 07:36:05,978 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-36d10166]) (logid:db30dace) Unable to start secondary storage VM [4245] for standby capacity, it will be recycled and will start a new one.
2026-02-04 07:36:06,450 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-36d10166]) (logid:db30dace) Revoving nic secondary ip entry ...
2026-02-04 07:36:06,461 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-36d10166]) (logid:db30dace) Revoving nic secondary ip entry ...
2026-02-04 07:36:06,478 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-36d10166]) (logid:db30dace) Revoving nic secondary ip entry ...
2026-02-04 07:36:06,598 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-36d10166]) (logid:db30dace) received secondary storage vm alert
2026-02-04 07:36:06,613 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-36d10166]) (logid:db30dace) Primary secondary storage is not even started, wait until next turn
2026-02-04 07:36:20,802 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0e01d6b3]) (logid:ce926fc6) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:36:24,608 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-ae653139]) (logid:cc276ddd) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:36:50,799 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:36:50,809 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) No running secondary storage vms found in datacenter id=11, starting one
2026-02-04 07:36:50,815 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Allocate secondary storage VM standby capacity for zone [11].
2026-02-04 07:36:50,820 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Found a stopped secondary storage VM instance {"id":4247,"instanceName":"s-4247-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"4ed37fd5-f9e2-410a-9401-933fd4b08bba"}, starting it.
2026-02-04 07:37:05,717 WARN [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Unable to start secondary storage VM [4247] due to [Unable to create a deployment for VM instance {"id":4247,"instanceName":"s-4247-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"4ed37fd5-f9e2-410a-9401-933fd4b08bba"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4247,"instanceName":"s-4247-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"4ed37fd5-f9e2-410a-9401-933fd4b08bba"}Scope=interface com.cloud.dc.DataCenter; id=11
2026-02-04 07:37:05,718 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Unable to start secondary storage VM [4247] for standby capacity, it will be recycled and will start a new one.
2026-02-04 07:37:06,463 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Revoving nic secondary ip entry ...
2026-02-04 07:37:06,475 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Revoving nic secondary ip entry ...
2026-02-04 07:37:06,492 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Revoving nic secondary ip entry ...
2026-02-04 07:37:06,608 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) received secondary storage vm alert
2026-02-04 07:37:06,621 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8f5eafe]) (logid:842aeee8) Primary secondary storage is not even started, wait until next turn
2026-02-04 07:37:27,655 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-07010cf5]) (logid:30877711) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:37:50,803 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:37:50,816 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) No running secondary storage vms found in datacenter id=11, starting one
2026-02-04 07:37:50,823 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Allocate secondary storage VM standby capacity for zone [11].
2026-02-04 07:37:50,830 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Found a stopped secondary storage VM instance {"id":4249,"instanceName":"s-4249-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"e5e4977e-8671-449e-82d1-aaaa6908657e"}, starting it.
2026-02-04 07:38:05,091 WARN [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Unable to start secondary storage VM [4249] due to [Unable to create a deployment for VM instance {"id":4249,"instanceName":"s-4249-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"e5e4977e-8671-449e-82d1-aaaa6908657e"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4249,"instanceName":"s-4249-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"e5e4977e-8671-449e-82d1-aaaa6908657e"}Scope=interface com.cloud.dc.DataCenter; id=11
2026-02-04 07:38:05,092 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Unable to start secondary storage VM [4249] for standby capacity, it will be recycled and will start a new one.
2026-02-04 07:38:08,201 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Revoving nic secondary ip entry ...
2026-02-04 07:38:08,212 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Revoving nic secondary ip entry ...
2026-02-04 07:38:08,227 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Revoving nic secondary ip entry ...
2026-02-04 07:38:08,333 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) received secondary storage vm alert
2026-02-04 07:38:08,345 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb8d956f]) (logid:b179b33d) Primary secondary storage is not even started, wait until next turn
2026-02-04 07:38:30,762 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-cadbea44]) (logid:795c746b) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:38:36,450 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-deea592f]) (logid:876359ff) Revoving nic secondary ip entry ...
2026-02-04 07:38:36,461 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-deea592f]) (logid:876359ff) Revoving nic secondary ip entry ...
2026-02-04 07:38:36,478 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-deea592f]) (logid:876359ff) Revoving nic secondary ip entry ...
2026-02-04 07:38:50,798 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:38:50,803 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) No running secondary storage vms found in datacenter id=11, starting one
2026-02-04 07:38:50,807 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Allocate secondary storage VM standby capacity for zone [11].
2026-02-04 07:38:50,811 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Found a stopped secondary storage VM instance {"id":4251,"instanceName":"s-4251-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"68bd545d-c33f-46a6-882f-6b5641be3cd5"}, starting it.
2026-02-04 07:39:06,116 WARN [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Unable to start secondary storage VM [4251] due to [Unable to create a deployment for VM instance {"id":4251,"instanceName":"s-4251-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"68bd545d-c33f-46a6-882f-6b5641be3cd5"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4251,"instanceName":"s-4251-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"68bd545d-c33f-46a6-882f-6b5641be3cd5"}Scope=interface com.cloud.dc.DataCenter; id=11
2026-02-04 07:39:06,118 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Unable to start secondary storage VM [4251] for standby capacity, it will be recycled and will start a new one.
2026-02-04 07:39:06,443 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-61973202]) (logid:769b663a) Revoving nic secondary ip entry ...
2026-02-04 07:39:06,453 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-61973202]) (logid:769b663a) Revoving nic secondary ip entry ...
2026-02-04 07:39:06,466 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-61973202]) (logid:769b663a) Revoving nic secondary ip entry ...
2026-02-04 07:39:06,571 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-61973202]) (logid:769b663a) received secondary storage vm alert
2026-02-04 07:39:06,583 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Primary secondary storage is not even started, wait until next turn
2026-02-04 07:39:20,795 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a37b87ec]) (logid:b5524b01) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:39:33,863 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-44b5f7bb]) (logid:8badfc6c) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:39:50,804 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ab63b95a]) (logid:0f9b19da) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:40:20,798 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c533ba5]) (logid:10df6fc4) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:40:36,443 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-4032de24]) (logid:40a2a447) Revoving nic secondary ip entry ...
2026-02-04 07:40:36,454 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-4032de24]) (logid:40a2a447) Revoving nic secondary ip entry ...
2026-02-04 07:40:36,470 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:[ctx-4032de24]) (logid:40a2a447) Revoving nic secondary ip entry ...
2026-02-04 07:40:37,037 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-7d956e8f]) (logid:495dfab8) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:41:20,805 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-458cca3c]) (logid:72c17630) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:41:39,912 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-77fcacc8]) (logid:9458926b) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:41:50,798 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-051a299b]) (logid:c917411d) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:42:20,799 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-44bbfa4d]) (logid:1cd61f89) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:42:42,647 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-eb4d29a6]) (logid:7a1beb07) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:42:55,628 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: secondary-storage-discoverer
2026-02-04 07:43:01,527 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/core/spring-services-secondary-storage-controller-core-context.xml]
2026-02-04 07:43:13,133 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (main:[]) (logid:) Start configuring secondary storage vm manager : PremiumSecondaryStorageManagerImpl
2026-02-04 07:43:17,753 INFO [c.c.a.SecondaryStorageVmAlertAdapter] (main:[]) (logid:) Start configuring secondary storage vm alert manager : SecondaryStorageVmAlertAdapter
2026-02-04 07:43:17,777 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.storage.secondary.SecondaryStorageVmDefaultAllocator@278e993c
2026-02-04 07:43:18,519 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [secondary-storage-discoverer] context.
2026-02-04 07:43:18,519 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [secondary-storage-discoverer] context.
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/secondary-storage-discoverer/spring-secondary-storage-discoverer-context.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml]
2026-02-04 07:43:18,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.20.2.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]
2026-02-04 07:43:18,541 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [secondary-storage-discoverer] in 22 ms
2026-02-04 07:43:18,931 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (main:[]) (logid:) Start secondary storage vm manager
2026-02-04 07:43:19,384 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-c745414f]) (logid:) Trying to obtain module [secondary-storage-discoverer] context.
2026-02-04 07:43:19,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-c745414f]) (logid:) Starting module [secondary-storage-discoverer].
2026-02-04 07:43:24,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-abdc4d1f]) (logid:d985168e) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2026-02-04 07:43:33,961 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-a1df3dfd]) (logid:2299c055) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:44:13,190 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7a5fb2e1]) (logid:a1374cee) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:44:37,052 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-2a9ef9ff]) (logid:d5a83bcc) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:44:43,170 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c0e3829]) (logid:3307e1d0) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:45:13,161 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-935e4ad9]) (logid:05c487de) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:45:39,846 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-ff2072a5]) (logid:483ed8aa) There is no secondary storage VM for secondary storage host Secondary01
2026-02-04 07:45:43,166 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Zone [11] is ready to launch secondary storage VM.
2026-02-04 07:45:43,180 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) No running secondary storage vms found in datacenter id=11, starting one
2026-02-04 07:45:43,188 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Allocate secondary storage VM standby capacity for zone [11].
2026-02-04 07:45:43,195 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Found a stopped secondary storage VM instance {"id":4259,"instanceName":"s-4259-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"ad0670e9-2205-44a7-8681-eae8c6deaabb"}, starting it.
2026-02-04 07:46:00,040 WARN [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Unable to start secondary storage VM [4259] due to [Unable to create a deployment for VM instance {"id":4259,"instanceName":"s-4259-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"ad0670e9-2205-44a7-8681-eae8c6deaabb"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4259,"instanceName":"s-4259-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"ad0670e9-2205-44a7-8681-eae8c6deaabb"}Scope=interface com.cloud.dc.DataCenter; id=11
2026-02-04 07:46:00,042 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Unable to start secondary storage VM [4259] for standby capacity, it will be recycled and will start a new one.
2026-02-04 07:46:01,020 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Revoving nic secondary ip entry ...
2026-02-04 07:46:01,027 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Revoving nic secondary ip entry ...
2026-02-04 07:46:01,036 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Revoving nic secondary ip entry ...
2026-02-04 07:46:01,141 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) received secondary storage vm alert
2026-02-04 07:46:01,149 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67106d9c]) (logid:3e7b1b67) Primary secondary storage is not even started, wait until next turn
root@mgmt1:~#

And also while extracting the template , this error comming

Error while extracting template. Extracting template: bca78a34-1afe-4fd2-9c5e-ff0241107052 from zone: a2226896-61b4-4954-98bd-ec88b350ae68

root@mgmt1:~# cat /var/log/cloudstack/management/management-server.log | grep ssvm
2026-02-04 06:03:31,323 ERROR [o.a.c.s.i.TemplateServiceImpl] (qtp1438988851-18:[ctx-69c2873e, ctx-06114b32]) (logid:aab3683f) No remote endpoint to send command, check if host or ssvm is down?
2026-02-04 06:06:57,427 WARN [c.c.v.SystemVmLoadScanner$1] (secstorage-1:[ctx-4cf32a5f]) (logid:a436bd47) Unexpected exception Cannot invoke "com.cloud.vm.SecondaryStorageVmVO.getUuid()" because "ssvm" is null java.lang.NullPointerException: Cannot invoke "com.cloud.vm.SecondaryStorageVmVO.getUuid()" because "ssvm" is null
2026-02-04 07:39:03,056 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (API-Job-Executor-1:[ctx-6b2cc03b, job-37430, ctx-0bcee463]) (logid:31b8a1ad) No remote endpoint to send command, check if host or ssvm is down?
2026-02-04 07:39:03,057 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (API-Job-Executor-1:[ctx-6b2cc03b, job-37430, ctx-0bcee463]) (logid:31b8a1ad) Unable to create a link for entity at template/tmpl/1/236/ on ssvm, No remote endpoint to send command, check if host or ssvm is down?
2026-02-04 07:39:03,074 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:[ctx-6b2cc03b, job-37430]) (logid:31b8a1ad) Unexpected exception while executing org.apache.cloudstack.api.command.user.template.ExtractTemplateCmd com.cloud.utils.exception.CloudRuntimeException: Unable to create a link for entity at template/tmpl/1/236/ on ssvm, No remote endpoint to send command, check if host or ssvm is down?
2026-02-04 07:39:03,080 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-6b2cc03b, job-37430]) (logid:31b8a1ad) Complete async job-37430, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Unable to create a link for entity at template/tmpl/1/236/ on ssvm, No remote endpoint to send command, check if host or ssvm is down?"}
2026-02-04 07:42:45,206 ERROR [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-105:[ctx-61dffb6e, job-37108/job-37478, ctx-ccca35df]) (logid:21211c35) No remote endpoint to send command, check if host or ssvm is down?
2026-02-04 07:42:45,221 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-105:[ctx-61dffb6e, job-37108/job-37478, ctx-ccca35df]) (logid:21211c35) Unable to create volume [{"name":"ROOT-4259","uuid":"58c62b0a-c0e7-4f59-96f3-6ac11fc47d19"}] due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-02-04 07:42:45,221 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-105:[ctx-61dffb6e, job-37108/job-37478, ctx-ccca35df]) (logid:21211c35) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:80] is unreachable: Unable to create volume [{"name":"ROOT-4259","uuid":"58c62b0a-c0e7-4f59-96f3-6ac11fc47d19"}] due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-02-04 07:42:45,221 DEBUG [c.c.d.D.ExcludeList] (Work-Job-Executor-105:[ctx-61dffb6e, job-37108/job-37478, ctx-ccca35df]) (logid:21211c35) Adding StoragePool [80] to the avoid set due to [Resource [StoragePool:80] is unreachable: Unable to create volume [{"name":"ROOT-4259","uuid":"58c62b0a-c0e7-4f59-96f3-6ac11fc47d19"}] due to [No remote endpoint to send command, check if host or ssvm is down?].].

[DaanHoogland]

the message

2026-02-04 07:39:06,116 WARN [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61973202]) (logid:769b663a) Unable to start secondary storage VM [4251] due to [Unable to create a deployment for VM instance {"id":4251,"instanceName":"s-4251-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"68bd545d-c33f-46a6-882f-6b5641be3cd5"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":4251,"instanceName":"s-4251-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"68bd545d-c33f-46a6-882f-6b5641be3cd5"}Scope=interface com.cloud.dc.DataCenter; id=11

seems the important one.

Is this a new installation or an upgraded one?

[muthukrishnang1100]

This is an Upgraded.

[DaanHoogland]

Please have a look at the logs during upgrading, did the template register, or are there any other exceptions in the log concerning the data migration.

[Muthukrish001]

Upgrading is completed 2 weeks before. After upgrading it will work fine. Last 3 days NFS server is down. So, the system VMs are starting and stopping continuously. We again remove and add NFS server again. But, still not work the system VMs. The System VM template not downloading correctly in secondary storage. The template size is displayed as 0 GB.