SDDC Manager deployment loop

[paulcotos]

Describe the bug

Once the VCF Installer is up and running and all images downloaded, the deployment starts.
Everything is OK with the vCenter deployment, but when it reaches the SDDC Manager it deploys, starts, waitsss for about 2 minutes and then deletes and redeploys the VM. It doess this on and on.
It finally fails with "SDDC Manager services failed to start." error.

I've retried to remove everything and sstart from scratch, but I get the same behaviour.

Reproduction steps

1. Run the full stack New-HoloDeckInstance command in holorouter.
2. Once the nested VCF 9 deployment starts, it doess not go past "Deploy SDDC Manager" step.

Expected behavior

VCF 9 full stack deployment without any isssues, as per documentation.

Additional context

No response

[dhruv-tyagi-broadcom]

Do you have enough physical resources? This kind of behavior generally happens when there aren't enough physical resources to power on the VM. You can SSH to VCF Installer and look at the logs at /var/log/vmware/vcf/domainmanager/domainmanager.log and look at the logs when the SDDC Manager VM is getting deployed and deleted.

[paulcotos]

Hello,
Thank you for the reply!
I did look into the log file and it does not seem to be an issue with resources.

It appears to be something related to an API invalid input or something. Below is the error and attached is the whole log file.

domainmanager.log

2025-07-25T10:35:58.635+0000 ERROR [vcf_dm,68835dd2249b0ce27eb377dd12dcec93,f721] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-21]  [B5K0QE] VCF_GUEST_SERVICE_SDDCMANAGER_WEBSERVICES_NOT_STARTED SDDC Manager services did not start. Check logs for more details.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: SDDC Manager services did not start. Check logs for more details.
	at com.vmware.evo.sddc.sddcmanager.DeploySddcManagerAction.postValidate(DeploySddcManagerAction.java:370)
	at com.vmware.evo.sddc.sddcmanager.DeploySddcManagerAction.postValidate(DeploySddcManagerAction.java:111)
	at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$1(FsmActionState.java:27)
	at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
	at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
	at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
	at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403)
	at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:556)
	at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
	at jdk.internal.reflect.GeneratedMethodAccessor265.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
	at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
	at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Retriable operation 'Verified that SDDC manager VM and web services are up and ready' failed to complete after 3 retries.
	at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:221)
	at com.vmware.evo.sddc.sddcmanager.DeploySddcManagerAction.postValidate(DeploySddcManagerAction.java:366)
	... 17 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_CERT_ALIAS_LOWERCASE"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_CERT_ALIAS_LOWERCASE"}],"referenceToken":"G6R88E"}"
	at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
	at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:672)
	at com.vmware.evo.sddc.sddcmanager.helper.CertificateConfigurationRestContractClient.setTrustedCertificatesConfig(CertificateConfigurationRestContractClient.java:77)
	at com.vmware.evo.sddc.sddcmanager.helper.SddcManagerUtil.setTrustedCertsAndRefresh(SddcManagerUtil.java:108)
	at com.vmware.evo.sddc.sddcmanager.helper.SddcManagerUtil.installCertificates(SddcManagerUtil.java:64)
	at com.vmware.evo.sddc.sddcmanager.DeploySddcManagerAction.importCertificate(DeploySddcManagerAction.java:566)
	at com.vmware.evo.sddc.sddcmanager.DeploySddcManagerAction.lambda$postValidate$0(DeploySddcManagerAction.java:353)
	at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:182)
	... 18 common frames omitted

[paulcotos]

So, I believe it's because the FQDN contains an uppercase A from "site-A.vcf.lab" which is comming from the holorouter script.
The commands to update the DNS do not work, I receive errors. Also, the add and delete DNS entries do not work. the add tells me that the entry exists and the remove one tells me that it does not exist.

Looking at the initial config file (I deleted everything, again), the domain is "null" everywhere.
Where can I actually change the "site-A"?

[paulcotos]

So, get this!
The uppercase A is the A or B that you put in the new deployment command.
Those need to be lowercase otherwise SDDC will not install because it does not accept upercase letters in it's FQDN (for some reason).

I had the -Site parameter set to A instead of a.

It feels kind of dumb if you ask me, considering I wasted a day and a half for something so meaningless.