[Bug]: Holodeck 9.0.2 with VCF 9.0.2.0 stalls at Install-VcfInstallerBundles — hardcoded -eq 7 mismatches API count of 8

[damir-topic]

Summary

Deploying VCF 9.0.2.0 (full VCF, not VVF) with Holodeck 9.0.2 stalls indefinitely at Install-VcfInstallerBundles. The function is gated on a hardcoded count of 7 in SddcMgmtDeployment.psm1, but VCF Installer 9.0.2.0's GET /v1/bundles/download-status?imageType=INSTALL returns 8 entries whose version matches 9\.0\.2\.0\.* once the depot is fully populated, so the loop never exits.

Symptom

The deployment log loops every 10 seconds:

SddcMgmtDomain[<pid>]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
SddcMgmtDomain[<pid>]: [INFO] Didn't receive all bundles. Received 8 bundle details. Trying again after 10 seconds

VCF Installer's depot UI (/vcf-installer-ui/portal/depot-home) shows all 7 visible components as Downloaded / Success:

• SDDC Manager 9.0.2.0
• VMware Cloud Foundation Automation 9.0.2.0
• VMware Cloud Foundation Operations 9.0.2.0
• VMware Cloud Foundation Operations Collector 9.0.2.0
• VMware Cloud Foundation Operations fleet management 9.0.2.0
• VMware NSX 9.0.2.0
• VMware vCenter 9.0.2.0

Restarting New-HoloDeckInstance does not help — the gate is count-based, not download-state-based, and the count Holodeck observes (8) never reaches the count it expects (7).

Root cause

/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 (the module bundled inside the HoloRouter OVA), Install-VcfInstallerBundles:

$bundle_details = Invoke-RestMethod -Uri "https://${HostName}/v1/bundles/download-status?imageType=INSTALL" ...

# ... isVVF / version branches ...
}elseif($Version -eq "9.0.2.0"){
    $vcf9_bundle_details = $bundle_details.elements | Where-Object {$_.version -match "9\.0\.2\.0\.*"}
}

# count gate:
if($isVVF -and ($Version -in "9.0.0.0", "9.0.1.0") -and ($vcf9_bundle_details.count -eq 6)){ ... }
elseif($isVVF -and ($Version -eq "9.0.2.0") -and ($vcf9_bundle_details.count -eq 3)){ ... }
elseif($vcf9_bundle_details.count -eq 7){                       # <-- the bug
    Write-Log -Message "Received all Bundle Details"
    $bundle_api_response = $true
}
else{
    Write-Log -Message "Didn't receive all bundles. Received $($vcf9_bundle_details.count) bundle details. Trying again after 10 seconds"
    Start-Sleep -Seconds 10
}

On VCF 9.0.2.0, the API returns 8 bundles matching 9\.0\.2\.0\.* (the 9.0.2 BOM split VCF Operations into Operations + Operations Collector + Operations Fleet Management — visible as separate rows in the depot UI plus one more entry in the API response, e.g. an additional VRA bundle). The hardcoded -eq 7 was the correct count for the 9.0.1 BOM and was not refreshed for 9.0.2.

Suggested fix

Change the non-VVF count check from equality to a lower-bound comparison so it tolerates the 9.0.1 count of 7, the 9.0.2 count of 8, and any future BOM split that increases the count without removing existing bundles:

 }elseif($isVVF -and ($Version -eq "9.0.2.0") -and ($vcf9_bundle_details.count -eq 3)){
     Write-Log -Message "Received all Bundle Details"
     $bundle_api_response = $true
-}elseif($vcf9_bundle_details.count -eq 7){
+}elseif($vcf9_bundle_details.count -ge 7){
     Write-Log -Message "Received all Bundle Details"
     $bundle_api_response = $true
 }

A version-conditional fix (-eq 8 for 9.0.2, -eq 7 for 9.0.0/9.0.1) would also work but reintroduces the same bug shape against the next BOM revision, so -ge 7 is preferred.

Workaround for affected users

Edit the module on the deployed HoloRouter:

sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
  /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1

Kill the in-flight pwsh (the running session holds the unpatched function in memory):

pkill -9 -f pwsh

Restart the deploy from a fresh pwsh:

Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> [<original flags>]

The state engine resumes at Install-VcfInstallerBundles (currently InProgress), the patched function exits the loop on first iteration, and the BUNDLE_DOWNLOAD_ALREADY_DOWNLOADED handler at the next try/catch block lets the download phase complete quickly.

Verified end-to-end on Holodeck 9.0.2 + VCF 9.0.2.0 / -ManagementOnly / online depot: after the patch the function logs Received all Bundle Details, all components report SUCCESS (with one extra bundle entry going through a real download), and the deploy advances to the management-domain phase.

Environment

• Holodeck: 9.0.2 (HoloRouter OVA build 9.0.2.0424)
• VCF Installer: 9.0.2.0
• Target VCF: 9.0.2.0 (full VCF, -ManagementOnly)
• Depot: online
• VCF Installer UI shows all 7 components Downloaded/Success; GET /v1/bundles/download-status?imageType=INSTALL returns 8 entries matching 9.0.2.0.*.

Note on issue tracking

Repo vmware/Holodeck ships only the docs site (MkDocs). The PowerShell module that contains the bug is bundled inside the HoloRouter OVA on the Broadcom Support Portal, not in this repo's source tree. Filing here per the Support page as the canonical issue tracker.

[damir-topic]

Update on the root cause after field-testing the workaround end-to-end (and reading more of the depot's API output):

The 9.0.2 GA BOM is in fact exactly 7 components — -eq 7 is correct for a clean depot. The 8th bundle in the field that triggers this deadlock is operator-introduced: a post-9.0.2-GA patch version of an existing component (in our case VCF Automation 9.0.2.0100.25300779) that the operator manually pulled into the depot alongside the GA bundle while troubleshooting a separate download issue.

The PowerShell regex 9\.0\.2\.0\.* (note: \.* = "zero or more literal dots") matches both the GA 9.0.2.0.25145732 and the patch 9.0.2.0100.25300779 because both strings start with 9.0.2.0. So GET /v1/bundles/download-status?imageType=INSTALL returns 8 entries instead of 7, and the loop never exits.

This makes the bug compound:

1. The hardcoded -eq 7 deadlocks Install-VcfInstallerBundles whenever the depot has any extra bundle for any reason (operator patch download, future BOM split, etc.). The -ge 7 patch in the issue body fixes this.
2. But the same offending patch bundle also fails VCF Installer's spec validator at the next phase with Component <X> version <patch> is a patch version that is not allowed for deployment. So -ge 7 alone unblocks the loop but the deploy still fails downstream — the operator also has to DELETE /v1/bundles/<patch-bundle-id> before the spec validator can pass.

Worth considering for the upstream fix: in addition to the -ge 7 count-check robustness, Install-VcfInstallerBundles could pre-filter $vcf9_bundle_details to exclude patch versions (e.g. tighten the regex to ^9\.0\.2\.0\.\d+$ rather than 9\.0\.2\.0\.*) so the count it observes matches what the spec validator will accept. That'd close both halves of the issue with one change.

For posterity, the depot DELETE we ran:

curl -ks -H "Authorization: Bearer $TOK" -H "Content-Type: application/json" \
  -X DELETE "https://<vcf-installer-fqdn>/v1/bundles/<offending-bundle-id>"
# returns HTTP 204 No Content on success

After the delete, VCF Installer's spec validator re-ran on its next polling cycle and passed; Holodeck's outer poll loop picked up the new state and the management-domain deploy proceeded.

[dhruv-tyagi-broadcom]

Thank you. That's a good find. We will fix this in the next release. Appreciate you digging into the code and finding the exact root cause and the workaround!