Cloudstack multiple management server connection across different networks

[arpanbht]

Hey guys, I was trying to make another management server in a different network. My initial cloudstack setup is running under 172.16.17.0/24 network and other cloudstack network will be 172.16.18.0/24.

I have changed the mysql bind-address to 0.0.0.0 in mgmt server 1 and in the db.properties in mgmt server 2 i have changed all the hosts to this management server 172.16.17.x the the mgmt server 1.

now this error is coming while connecting to the 1st mgmt server's mysql database

2026-08-19 08:48:43,916 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) module start failure of module [backup] was due to: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.22.1.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]: Cannot resolve reference to bean 'DefaultConfigResources' while setting bean property 'locations'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:113)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1707)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:171)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:748)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:174)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:143)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:274)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:262)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:131)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
        at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
        at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:53)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:204)
        at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:113)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:874)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1358)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:309)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:330)
        ... 55 more

Also I edited the management cidr for production to 172.16.0.0/16 (contains both 172.16.17.0/24 and 172.16.18.0/24) and for testbed to 192.168.0.0/16.

I have made a whole setup seperately in a test bed with two management server, there both the management servers are getting connected but both the management servers are in same network 192.168.1.0/24.

Please help me with this matter.

[weizhouapache]

Also I edited the management cidr for production to 172.16.0.0/16 (contains both 172.16.17.0/24 and 172.16.18.0/24) and for testbed to 192.168.0.0/16.

did not help ?

[arpanbht]

@weizhouapache @DaanHoogland

Previously on the 1st management server I have setup the database usign this command

sudo cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root

But for the new manaement server I have used something like this

cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root:rootpassword -e file -m MgmtSecretKey123 -k DbSecretKey123 -i 192.168.1.240

Can it create any problem? As the management server key which is in /etc/cloudstack/management/key is different on both the management server.

[weizhouapache]

@weizhouapache @DaanHoogland

Previously on the 1st management server I have setup the database usign this command

sudo cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root

But for the new manaement server I have used something like this

cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root:rootpassword -e file -m MgmtSecretKey123 -k DbSecretKey123 -i 192.168.1.240

Can it create any problem? As the management server key which is in /etc/cloudstack/management/key is different on both the management server.

yes, the database key must be the same in the two commands
if not specified, the default mgmt key and database key are "password"

moreover, deploy the db only once on the first management server (--deploy-as=root)

[arpanbht]

@weizhouapache so i don't need to setup the database again in the 2nd management server?

Tell me the steps which i need to follow in order to setup the new management server so both the management servers can be accessible

[weizhouapache]

@weizhouapache so i don't need to setup the database again in the 2nd management server?

correct.

Tell me the steps which i need to follow in order to setup the new management server so both the management servers can be accessible

refer to https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html#additional-management-servers

[arpanbht]

@weizhouapache
Ok I will try that and if i face any kind of issues I will reach out to you.

[RyanP-Testing]

Hi Arpan,

Thanks for sharing the detailed configuration and logs. We went through the issue carefully, and based on the behavior you described, we don't believe the fact that the two Management Servers are on different subnets is, by itself, the problem. CloudStack can run multiple Management Servers against the same CloudStack database, even when the servers are on different routed networks.

The important point here is to identify what changes when MGMT-02 moves from the same subnet to 172.16.18.0/24.

The DefaultConfigResources exception is occurring during CloudStack's Spring/module initialization, so we would not immediately treat it as a direct MySQL connectivity error. It may be a secondary error caused by an earlier configuration or initialization problem.

We recommend checking the following in order:

1. Verify both Management Servers are using the same CloudStack database

MGMT-02 should be pointing to the existing CloudStack database used by MGMT-01. Please make sure the database was not initialized separately on MGMT-02.

Also, please review the db.properties configuration on both servers and compare the database-related entries.

2. Check the Management Server encryption key

You mentioned that /etc/cloudstack/management/key is different on the two Management Servers. This is something we would definitely investigate.

Please compare the keys on both servers:

sha256sum /etc/cloudstack/management/key

For security, please don't share the actual key publicly. If the hashes are different, we recommend correcting the configuration so both Management Servers use the appropriate shared key for the same CloudStack deployment.

3. Verify connectivity from MGMT-02 to the database

From the 172.16.18.x Management Server, test:

nc -vz 172.16.17.x 3306

If that succeeds, also verify actual MySQL authentication:

mysql -h 172.16.17.x -u cloud -p

Changing MySQL's bind-address to 0.0.0.0 only makes MySQL listen on the available interfaces; the MySQL user grants and firewall rules must also allow the remote Management Server.

On MGMT-01, you can verify that MySQL is listening with:

ss -lntp | grep 3306

4. Verify routing and firewall rules between the two networks

The change to:

172.16.0.0/16

is fine from a CIDR perspective because it covers both 172.16.17.0/24 and 172.16.18.0/24.

However, the Management CIDR setting itself does not create routing between the networks.

Please make sure there is an actual route and that the required traffic is permitted in both directions between:

172.16.17.0/24
        ↕
   Router / Firewall
        ↕
172.16.18.0/24

5. Use the same-subnet test as a controlled comparison

Since you already confirmed that two Management Servers work when both are on 192.168.1.0/24, that is actually a very useful test result.

If possible, perform one more controlled test with both Management Servers temporarily placed on the same 172.16.17.0/24 network while using the same database and configuration.

If that works, and moving MGMT-02 back to 172.16.18.0/24 causes the failure again, we can confidently focus on the inter-subnet routing, firewall, MySQL remote access, or related network configuration rather than CloudStack's basic multi-Management-Server functionality.

6. Check the first error in the Management Server log

The DefaultConfigResources exception may be the final result of an earlier failure.

Please check the Management Server log immediately before this exception and look for the first occurrence of:

ERROR
Exception
MySQL
database
connection
encryption
key

That first error will usually tell us much more about the actual root cause.

What we would check first

Based on the information currently available, our priority would be:

1. Confirm both Management Servers use the same CloudStack database.
2. Verify the Management Server/encryption keys are consistent.
3. Confirm MGMT-02 can reach MySQL on TCP/3306.
4. Verify MySQL grants and firewall rules.
5. Verify routing between 172.16.17.0/24 and 172.16.18.0/24.
6. Compare the relevant CloudStack configuration between both nodes.
7. Then investigate the DefaultConfigResources exception further if the above checks are clean.

We would avoid rebuilding the environment or making further configuration changes until these checks are completed. The fact that the same setup works when both Management Servers are on the same subnet gives us a good baseline, and we can use that to isolate the exact point of failure rather than changing multiple variables at once.

If you can share the sanitized db.properties from both Management Servers, the SHA-256 hashes of the management keys, and the result of the TCP/3306 connectivity test from MGMT-02, we can narrow this down considerably.

Regards,

**DevOps Engineer & Technical Support Engineer **
AccuWeb.Cloud