Control plane down ~3h, auto-recovery looping with ClusterConfigureError (deloryx-prod-1, DFW)

[dsolis-deloryx]

Hi team 👋

Our production control plane went down this afternoon and the automatic recovery doesn't seem to be able to bring it back.

Details

• Organization: deloryx
• Cloudspace: deloryx-prod-1
• Region: us-central-dfw-1 (gen2)
• Control plane endpoint: hcp-1b17d935-3252-4cc5-b2da-0bb27c7d0e9f.spot.rackspace.com — TLS handshake timeout on every request since ~16:30 UTC (2026-07-03)

What we observe

• The Control Plane Memory chart in the console shows API Server 1 running steadily for ~48h, then dying at ~16:30 UTC — followed by short-lived API Server 2 … API Server 10 blips, so the replacements appear to be crash-looping.
• Cloudspace status cycles between Error / Provisioning with reasons ControlPlaneUnresponsive, ControlPlaneConfigureFailed and ClusterConfigureError (e.g. "error while creating storage classes … TLS handshake timeout", "error creating/updating OIDC ClusterRoleBinding …").
• The console is stuck at "Deploying Cloudspace — 75%, Cluster Status Ready: In Progress" (similar to Stuck at "Deploying Cloudspace app" for a existing cloudspace #387).

Impact

• All 16 worker nodes are healthy and most workloads keep serving, but anything needing the API (database failover, pod rescheduling, GitOps) is blocked, and a few production services are down as a result.
• No changes on our side line up with the failure — the API server showed flat CPU/memory right up to the moment it died.

Could you take a look at the control plane host / recycle the control plane for us? Happy to provide HAR captures or any other diagnostics. Thanks a lot! 🙏

[dsolis-deloryx]

Update after ~9h: the incident has escalated from control-plane-only to a full customer-facing outage.

• Cloudspace status progressed ControlPlaneUnresponsive → ClusterConfigureError → now ServerConfigureFailed
• The control plane briefly answered around 02:25 UTC, then went unreachable again
• Since ~02:30 UTC the worker data plane is also down: our site returns Cloudflare 525 (origin TLS handshake failure) and all LoadBalancer-backed endpoints fail TLS — previously workloads kept serving throughout

It looks like the recovery pipeline is now reconfiguring the worker nodes themselves, which has taken down ingress. Production is fully offline. Could someone from the team take a look urgently? Org deloryx, cloudspace deloryx-prod-1, us-central-dfw-1. Thanks 🙏

[dsolis-deloryx]

Diagnostic detail that may help (~15h in): the API server now starts every ~30-60 min and serves /livez = 200 for 60-90 s each time (observed 04:13, 04:51, 05:27, 05:40, 06:06, 06:42, 07:42 UTC), but authenticated API requests always fail during these windows — the auth/data path never comes up before it dies again. That pattern points at the control-plane etcd/datastore rather than the API server process itself (consistent with your issue #386 on control-plane database growth).

Worker data plane recovered on its own around 04:51 and is holding (site + registry serving), but our database failover chain remains blocked without a working API, so parts of production are still down.

Could the team check the control plane's datastore for deloryx-prod-1? Thanks!

[dsolis-deloryx]

Resolved — for the record: the control plane came back and has been stable since ~08:11 UTC (authenticated API calls succeeding; kubectl fully functional since ~08:15; cloudspace status now Ready). Total outage ≈ 16 hours (16:30 UTC Jul 3 → 08:15 UTC Jul 4), including ~6 hours where the worker data plane / ingress was also down.

Post-recovery we had to manually clear stale kube-proxy state on all nodes (services got connection refused to healthy ClusterIPs after the long API blackout) and re-sync an OpenSearch security config that a job failed to apply mid-outage — worth noting for anyone else recovering from a long control-plane loss.

Two asks for the team when you see this:

1. A root-cause summary for the ~16h control-plane loss (the API server crash-looped ~10 restarts, then /livez flapped every 30-60 min with the auth/data path never coming up — looked like the datastore, ref Detect K8s control plane databases that are growing out of bounds and need to be compacted #386).
2. We're planning to enable the HA control plane now — would that have avoided this failure mode, or was the datastore shared?

Thanks!

[vjha-pf9]

@dsolis-deloryx I have checked your Cloudspace and nodes, both looks healthy.