Cluster upgrade stuck 1.33.0 → 1.34.9 across multiple cloudspaces — node version conflict blocking recycle (same signature as #324, #452)

[jedarden]

We're seeing the same pattern reported in #452 and #324 on three cloudspaces during the 1.33.0 → 1.34.9 upgrade:

• apexalgo-iad (hcp-99476ebb-4133-4a21-ac6a-6e2bdf6794c0)
• iad-ci (hcp-de5bec10-ce14-4eed-a6f4-750f3fd3a89a)
• ord-devimprint (hcp-5f30c973-cde7-42d9-8c7b-5d0573821330)

Three other cloudspaces in the same org (rs-manager, iad-kalshi, iad-options) completed the identical upgrade cleanly, so this looks isolated to a subset of nodes/cloudspaces rather than a blanket rollout failure.

On the affected cloudspaces, new nodes created during the rollout register, send exactly one heartbeat, then flip to NotReady/Unknown ~60-75s after creation and never recover — stuck 8+ hours now.

Manually recycling the stuck node in iad-ci fails with:

admission webhook "vcloudspace.ngpc.rxt.io" denied the request: newVersion 1.33.0 is not allowed for oldVersion 1.34.9

which matches the version-conflict block described in #324.

apexalgo-iad additionally had a separate, previously-healthy 32-day-old node drop to NotReady mid-upgrade, suggesting some broader instability during this rollout window on that cloudspace, not just new-node bootstrap failures.

Could someone take a look, similar to how #324 was resolved (recycling the bad node server-side)? Happy to share more diagnostics.

[vjha-pf9]

@jedarden Thank you for reporting this issue; we’ll look into it.

[vjha-pf9]

@jedarden Cluster is healthy now, please check and confirm.

[jedarden]

Confirmed — all three cloudspaces (apexalgo-iad, iad-ci, ord-devimprint) are showing every node Ready on v1.34.9 now, no version mismatches left. Thanks for the quick turnaround on this!