URGENT: Admission webhook down — all VM create operations returning 500

[elijahdev0]

All VM resource creation (VMCloudSpace, VMPool) is failing with the same error:

Internal error occurred: failed calling webhook "vvmcloudspace.ngpc.rxt.io":
failed to call webhook:
Post "https://admission-webhook-service.ngpc.svc.cluster.local/validate-ngpc-rxt-io-v1-vmcloudspace?timeout=10s":
dial tcp 10.21.216.106:443: connect: connection refused

And for VMPool:

Internal error occurred: failed calling webhook "vvmpool.ngpc.rxt.io":
failed to call webhook:
Post "https://admission-webhook-service.ngpc.svc.cluster.local/validate-ngpc-rxt-io-v1-vmpool?timeout=10s":
dial tcp 10.21.216.106:443: connect: connection refused

The admission webhook pod at 10.21.216.106:443 is refusing connections. Reads (GET) and deletes (DELETE) still work. Only creates (POST) are affected. Existing resources are running fine but stuck in Terminating when deleted. Org: org-nlbbp4cczaekezhm.

[elijahdev0]

My SaaS is unusable because of this please help!

[elijahdev0]

@dramaturg

[vjha-pf9]

@elijahdev0 Service is working normally now at our end, please check now.

[dramaturg]

Hey Elijah,

Just FYI I'm not part of the team.

Have a nice day 🙃

[elijahdev0]

@elijahdev0 Service is working normally now at our end, please check now.

its down again!

[willkelly]

This incident appears to be recurring as of 2026-07-12 12:00 UTC. VMCloudSpace creation in org wilk-dev fails consistently through both the hosted MCP and the public Spot API with HTTP 500:

failed calling webhook "vvmcloudspace.ngpc.rxt.io": Post "https://admission-webhook-service.ngpc.svc.cluster.local/validate-ngpc-rxt-io-v1-vmcloudspace?timeout=10s": x509: certificate signed by unknown authority

Read and VMSSHKey operations work; VMCloudSpace admission is the failing boundary. Guarded retries have reproduced this for several hours without creating a resource. Could the admission webhook CA bundle/certificate be repaired again?

[vjha-pf9]

Thanks for reporting. This issue is fixed now.
Please let us know if you are still observing this problem.

[willkelly]

Can confirm it works!