Cloudspace stuck in Provisioning/ServerConfiguring for 70-150 min with node pool already Fulfilled (us-east-iad-1)

[virtusapps]

Hi team,

Same org as #457 (org_5fVi0IXjUr16TWP6). The SpeedBump issue there was resolved, thanks. This is a different one: the node pool is Fulfilled and the cloudspace never leaves Provisioning / ServerConfiguring, so the machine never joins the cluster.

Region: us-east-iad-1. Cloudspace name: channels-veo-render (recreated between attempts).

Attempt 1 — 2026-08-14

event	time (UTC)
cloudspace created	20:57:52
node pool 828ee02a-27c4-43bd-b78b-a4b481d956c6 created, Fulfilled, wonCount 1	21:01:50
first node actually Ready in kubectl	~22:11 (~70 min after the bid was fulfilled)

The whole time the API reported status: Provisioning, message: ServerConfiguring. kubectl get nodes returned nothing until 22:11. Pods that had been created 70 min earlier (coredns, ceph-csi-rbd-provisioner, metrics-server, headlamp) sat in ContainerCreating for that entire window, since there was no node to schedule them on.

Attempt 2 — 2026-08-15 (open right now)

event	time (UTC)
cloudspace created (endpoint hcp-7c14ea7f-95eb-4455-a494-ae92110d343c)	22:53:15
node pool 56da8adb-e286-411a-803b-0acb2f6b9ecf, Fulfilled, wonCount 1	22:58:02
kubectl get nodes	still No resources found at 23:27 (29 min and counting)

Before attempt 1 we had another run that sat in the same state for over 150 min before we tore it down.

This does not look like the auction. For ch.vs1.2xlarge-iad the API currently reports minBidPricePerHour $0.035 and currentMarketPricePerHour $0.010; our bid is $0.190, and the pool reports status: Fulfilled with wonCount 1 in every attempt. The bid is won — the server just never finishes configuring.

Questions:

1. Is there something stuck on the server-configuring side for this org/region that you can clear? tired from rackspace, a lot of problems, nodes provisioning for 2 days #461 was described as "stuck in serverconfiguring because of pending bidding" and needed manual action on your side — is this the same class of problem?
2. Is there any signal we can poll to distinguish "still configuring normally" from "wedged"? Today Provisioning/ServerConfiguring looks identical at minute 5 and at minute 150, so our supervisor cannot tell whether to keep waiting or destroy and recreate.
3. Does destroying and recreating the cloudspace help, or does it put us at the back of the same queue? We have done it three times now and each attempt started the clock over.

We run batch video rendering, so a cloudspace that takes 1-2h to hand over its first node effectively means the whole batch misses its window. Happy to leave the current cloudspace untouched if that helps you inspect it live — just let us know.

Thanks!

[vjha-pf9]

Seems like you have deleted the Cloudspace. We are unable to find the Cloudspace with the name channels-veo-render.

[virtusapps]

Thanks for looking. To answer directly: yes, we had deleted it while debugging — we've since changed our own approach to stop destroying the cloudspace automatically (we now scale node pools to zero/back up instead, keeping the cloudspace itself alive), specifically to stop hitting this class of issue on cloudspace creation. channels-veo-render (us-east-iad-1) exists again now, recreated 2026-08-21 15:38:47Z.

That change surfaced a related-but-distinct symptom, which is what I want to report here: the same "won the bid, node never joins" behavior also happens when adding a new node pool to a cloudspace that has been up and Ready for hours — not just at cloudspace creation.

Concrete data from today (2026-08-21/22), same org (org_5fVi0IXjUr16TWP6):

channels-veo-render (us-east-iad-1, up since 15:38:47Z that day):

• Pool e475cc45-170a-4210-af96-946f12acf47e (5 machines, bid $0.10), a39628a6-4e66-4b45-9f9d-bcd2517c5f58 (2, $0.10), 4b39a2f5-966c-4272-8b91-48b90c067667 (5, $0.10) — all created ~01:20 UTC on the 22nd, all reported status: Fulfilled with wonCount == desired within a couple minutes. 47+ minutes later, zero of those 12 machines had ever registered as a Kubernetes node (kubectl get nodes only ever showed the pre-existing node from an older pool). We deleted them just now.
• Earlier that same cloudspace, 3 more pools (ca449bdb…, 341abb9b…, f3178fc3…, 14 machines total, bid $0.04, created 23:29–00:10 UTC) had the identical symptom for 1h40m+ before we gave up and deleted them.

channels-veo-render-dfw (us-central-dfw-1, created 23:30:07Z): pool 68cfa9b8-e216-4fd3-87cb-d8585fa6237e (1 machine, bid $0.04) — Fulfilled, wonCount: 1, zero nodes for 2.5 hours. Deleted just now.

For contrast, in the same account/day we also stood up channels-veo-render-ord (us-central-ord-1): the very first pool there (bid $0.10) got a working node in ~4 minutes, and three more pools added minutes later (bid $0.04 this time) delivered 11 working nodes within 10–20 minutes each — completely normal.

So: same account, same day, bid prices ranging $0.04–$0.10 (all comfortably above that day's market, which swung $0.01–$0.11), one region/cloudspace pair (iad + dfw) stuck for hours on end while a third (ord) behaved exactly as expected. That rules out bid price as the cause for us, and doesn't look account-wide either — it looks isolated to specific cloudspaces/regions going into some bad state where the bid auction bookkeeping (Fulfilled/wonCount) completes but the actual VM never finishes joining Kubernetes.

Given channels-veo-render is a cloudspace we now intend to keep running long-term (not recreate), is there anything on your side that can look at why pools e475cc45/a39628a6/4b39a2f5/68cfa9b8 never delivered real machines, even though the bid was won? Happy to leave a fresh reproduction up if that helps — just let us know before we clean it up next time.

[vjha-pf9]

@virtusapps We can see only channels-veo-render-ord has 16 nodes available, while the other two cloudspaces channels-veo-render-dfw and channels-veo-render don’t have any node pool as of now. Also, all 3 cloudspaces are healthy.