Control plane shows healthy but none of the pods are being scheduled. Status page shows it's down #262
Replies: 8 comments 29 replies
|
@hariprasadiit what's the name of the Cloudspace (or Cloudspaces) that were affected? |
|
the cluster has been unreliable for the past couple of days and giving the impression that we shouldn't use it even for beta. |
|
@hariprasadiit your cluster has been having unusual issues where the kube scheduler gets into a bad state, failing to talk to the API scheduler. We have manually gotten this unstuck again. Is there anything unusual you are running in your cloudspace, since we don't see this error anywhere else? Anything that would interfere with the functioning of the kube scheduler or interact with the kube scheduler in unusual ways? |
|
We've Rancher agent deployed to monitor and manage the cluster. Except that, we don't have any other service that may interact with scheduler. I've couple of cluster from other cloud providers connected to the Rancher and none of them have this issue. |
|
@hariprasadiit thanks for your patience. We'll have to file and track this as a new issue. It may take us some time to get to it but rest assured we will chase it down. Please keep the Cloudspace running (we're happy to issue a usage credit). For reference, here's the error we are seeing in the control plane: |
|
Filed as #268 |
|
@sirishkr This issue is back now since today morning. Please help us resolve this. No new pods are being scheduled and have been in pending state |
|
Could you pls share cloudspace name and Org details ? |
|
The control plane has been restarted because of multiple timeout entries from calcio and also due to a kubelet issue with several nodes on the same day. |
|
@vjha-pf9 I don't know if it's Spot itself is not production ready or it's my cluster, it's pretty unstable. Some times pods are not being scheduled, some times they're being evicted, some times intermittent cluster networking issues causing services to take long time to load |
|
is there any that can be done to make this cluster reliable? |
|
@vjha-pf9 I recycled nodes and created a new node pool, now they're not being created at all. 0 nodes now. Please help urgently |
|
@sirishkr your help is appreciated here |
|
We’ve checked your cloudspace and it looks like you’ve lost the bid. You could try a different flavour or increase your bid price. |
|
@vjha-pf9 thanks for the update. I tried $10/hr without success. it shows unfulfilled. Do I need try more? I never saw it crossing $0.1/hr before |
|
We have checked further and found that the payment method is not available for your org. |
|
@vjha-pf9 thanks for the pointing in the right direction. I fixed billing issue. Now nodes are stuck in |
|
Please check now, we have fixed your cloudspace. |
|
@vjha-pf9 the issue is back. pods are not being scheduled now. Could you please fix this permanently? or please give us an option to restart the control plane |
|
@hariprasadiit Cloudspace looks healthy, please check and confirm. |
|
It's working now. Thanks for the help |
|
Hi @vjha-pf9 Please help resolve this. |
|
Hi @vjha-pf9 Cloudspace: beta Please help |
|
Please try a different flavor as this one is taking longer than usual to provision. |
|
@vjha-pf9 There seems to be some issue with the cluster. Please restart the control plane. some times it's not scheduling pods and some times pods are stuck in terminating state. Cloudspace: beta |
|
Cloudspace is healthy now. Please check the status. |
|
seems better now. Thank you |
Uh oh!
There was an error while loading. Please reload this page.
Hello,
Control plane is down since this morning resulting in none of the pods being scheduled.
When will this be resolved? even HA control plane is down.
All reactions