New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cluster hung up starting, waiting for etcd volumes to attach #4844
Comments
I solved the issue. Apparently, the new M5 AWS instance types are not supported in kops 1.8.1, as it does not support NVME for EBS volumes yet. Changing the instance type to M4 resolved the issue. There should be a warning when attempting to use unsupported instances types when provisioning a kops cluster. |
Still same issue with M5 on kops 1.9.0 |
Are you using Debian as your base image? If so, I assume Jessie? If you upgrade to Stretch that resolves the issue and allows you to use M5 with 1.9.0. |
@thereverendtom I have kops 1.9.0 installed and cluster upgrade has nothing to do as it is already 1.9.3 from stable. It's still Jessie and from what I can tell from channels (stable, alpha) there is no Stretch option. So as kops 1.9.0 stands with deploying 1.9.3 it doesn't support M5 without some manual changes as far as I can tell because |
Yeah, I think you may have to make a manual edit to the cluster config to specify Stretch over Jessie. I have kops generating Terraform, and then using that, so I just made the manual changes before applying the Terraform. |
------------- BUG REPORT TEMPLATE --------------------
kops
version are you running? The commandkops version
, will displaythis information.
1.8.1
kubectl version
will print theversion if a cluster is running or provide the Kubernetes version specified as
a
kops
flag.1.8.6
AWS
I provisioned a cluster using kops-generated Terraform, that I modified somewhat to work with the rest of my infrastructure. The cluster was been running for weeks without issue.
Yesterday, I tore down the cluster and rebuilt it (terraform destroy/apply). The cluster will not come back up. Instead, protokube hangs, waiting for the etcd volumes to attach. Here is the log output. The "waiting for volume to be attached" message repeats endlessly. I have confirmed via AWS CLI and console that the EBS volume is attached to the EC2 instance.
The cluster should start normally.
kops get --name my.example.com -oyaml
to display your cluster manifest.You may want to remove your cluster name and other sensitive information.
Please run the commands with most verbose logging by adding the
-v 10
flag.Paste the logs into this report, or in a gist and provide the gist link here.
Anything else do we need to know?
The text was updated successfully, but these errors were encountered: