Apache CloudStack Multi-Management Server Across Different Subnets with Shared MySQL DB – HA and Cross-Zone Snapshot Copy Issues #13220

arpanbht · 2026-05-22T10:56:56Z

arpanbht
May 22, 2026

Hi everyone,

I am currently testing a multi-management-server Apache CloudStack setup across two separate private networks and wanted clarification regarding management server failover behavior and a cross-zone snapshot/template copy issue.

Current Architecture

I have two separate networks:

192.168.1.0/24
192.168.2.0/24

Both networks are reachable from each other.

Infrastructure Layout

Network 1 (192.168.1.0/24)

Management Server - 192.168.1.101
KVM Host - 192.168.1.102
NFS Secondary Storage - 192.168.1.103

Network 2 (192.168.2.0/24)

Management Server - 192.168.2.101
KVM Host - 192.168.2.105
NFS Secondary Storage - 192.168.2.102

Setup Process

Initially, I deployed the entire CloudStack infrastructure in the 192.168.1.0/24 network.
Later, I created another standalone management server in the 192.168.2.0/24 network.
Instead of using a separate database, I pointed the second management server to the same MySQL DB used by the first management server.
I changed the MySQL configuration:
bind-address = 0.0.0.0

so both management servers can connect to the same DB.

I configured:
management.network.cidr=192.168.0.0/16

to allow both subnets to communicate within the management network range.

Then, Created another Zone
```
Added the 192.168.2.x KVM host
```

Added the second NFS server as secondary storage for that zone

Current Behavior Working Properly

Both zones are operational
All 4 System VMs are running properly
Hosts can communicate across subnets
Shared database connectivity works
Both management servers are operational

Observed Concern

Even though I have two management servers, all KVM hosts appear to connect only to the first management server (192.168.1.101) which also hosts the MySQL DB.

I want to understand:

HA / Failover Question

If the first management server (192.168.1.101) goes down:

Will the hosts automatically reconnect to the second management server (192.168.2.101)?
Will CloudStack continue functioning normally?
Is additional HA or load balancer configuration required?
Does CloudStack management server failover work automatically when multiple management servers share the same DB?

Cross-Zone Snapshot/Template Copy Problem

Everything works until I try to:

Copy templates across zones
Copy VM snapshots across zones

Then I receive:

HTTP Server returned 403 (expected 200 OK)

Relevant Logs

Failed to copy snapshot: 9d3f5a15-c8f8-4c00-9d08-1854ba68a90e with error:
HTTP Server returned 403 (expected 200 OK)

Additional stack trace:

[Failed to copy snapshot: 9d3f5a15-c8f8-4c00-9d08-1854ba68a90e with error:  HTTP Server returned 403 (expected 200 OK) ].
2026-05-22 06:10:04,241 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:[ctx-ce32d305]) (logid:93c14a4e) Failed to copy snapshot: 9d3f5a15-c8f8-4c00-9d08-1854ba68a90e with error:  HTTP Server returned 403 (expected 200 OK)
2026-05-22 06:10:04,250 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:[ctx-d6dd543b, job-571]) (logid:c12eb79c) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.CopySnapshotCmd com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.copySnapshot(SnapshotManagerImpl.java:2315)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy244.copySnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CopySnapshotCmd.execute(CopySnapshotCmd.java:188)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Questions Regarding the 403 Error

Is this issue related to:

Secondary storage permissions?
Apache/Nginx configuration?
SSVM-to-secondary-storage communication?
Cross-zone NFS accessibility?

Do both secondary storage servers need:

Mutual access?
Proper HTTP export configuration?
Additional ACL or firewall rules?

Since both zones are in different subnets, is there any additional configuration needed for:

SSVM routing
Secondary storage access
Template/snapshot copy traffic

Is the management.network.cidr=192.168.0.0/16 configuration sufficient for this type of architecture?

Additional Notes

Both subnets can ping each other
System VMs are running in both zones
KVM hosts are healthy
Secondary storage mounts work locally
Shared MySQL DB works from both management servers

I would appreciate guidance on:

Proper multi-management-server HA behavior
Best practices for multi-subnet CloudStack deployments
Root cause of the HTTP 403 during cross-zone snapshot/template copy

DaanHoogland · 2026-05-22T11:30:14Z

DaanHoogland
May 22, 2026
Collaborator

If the first management server (192.168.1.101) goes down:

Will the hosts automatically reconnect to the second management server (192.168.2.101)?

yes

Will CloudStack continue functioning normally?

yes

Is additional HA or load balancer configuration required?

only if there are external machines that need access to the UI without interuption and not caring about which MS they reach.

Does CloudStack management server failover work automatically when multiple management servers share the same DB?

MSses devide the work amongst them selves. jobs that are ongoing on an MS may fail if that machine crashes, but overall operations continue. The DB should not be an issue if the MSses are connected to a running engine.

I’ll try to look at your other questions later, but have no idea when I’ll take the time for that (sorry about that)

3 replies

arpanbht May 22, 2026
Author

@DaanHoogland Okay thank you for your reply, I was just confirming if the current architecture is working fine or not.

just tell me the below error when i try to copy snapshots and template across the zones

here is the error message I am getting on the management server console

[Failed to copy snapshot: 9d3f5a15-c8f8-4c00-9d08-1854ba68a90e with error:  HTTP Server returned 403 (expected 200 OK) ].
2026-05-22 06:10:04,241 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:[ctx-ce32d305]) (logid:93c14a4e) Failed to copy snapshot: 9d3f5a15-c8f8-4c00-9d08-1854ba68a90e with error:  HTTP Server returned 403 (expected 200 OK)
2026-05-22 06:10:04,250 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:[ctx-d6dd543b, job-571]) (logid:c12eb79c) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.CopySnapshotCmd com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot
        at com.cloud.storage.snapshot.SnapshotManagerImpl.copySnapshot(SnapshotManagerImpl.java:2315)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy244.copySnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CopySnapshotCmd.execute(CopySnapshotCmd.java:188)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

DaanHoogland May 22, 2026
Collaborator

does not seem right, what version are you using?

arpanbht May 22, 2026
Author

cloudstack version is 4.22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apache CloudStack Multi-Management Server Across Different Subnets with Shared MySQL DB – HA and Cross-Zone Snapshot Copy Issues #13220

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Apache CloudStack Multi-Management Server Across Different Subnets with Shared MySQL DB – HA and Cross-Zone Snapshot Copy Issues #13220

Uh oh!

arpanbht May 22, 2026

Current Architecture

Infrastructure Layout

Network 1 (192.168.1.0/24)

Network 2 (192.168.2.0/24)

Setup Process

Current Behavior Working Properly

Observed Concern

HA / Failover Question

Cross-Zone Snapshot/Template Copy Problem

Relevant Logs

Questions Regarding the 403 Error

Additional Notes

Replies: 1 comment · 3 replies

Uh oh!

DaanHoogland May 22, 2026 Collaborator

Uh oh!

arpanbht May 22, 2026 Author

Uh oh!

DaanHoogland May 22, 2026 Collaborator

Uh oh!

arpanbht May 22, 2026 Author

arpanbht
May 22, 2026

Replies: 1 comment 3 replies

DaanHoogland
May 22, 2026
Collaborator

arpanbht May 22, 2026
Author

DaanHoogland May 22, 2026
Collaborator

arpanbht May 22, 2026
Author