unable to change zones of control plane

[schollii]

1. What kops version are you running? The command kops version, will display
this information.

1.13

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.13

3. What cloud provider are you using?

aws

4. What commands did you run? What is the simplest way to reproduce this issue?

Create cluster:

1. kops create cluster --name ABC --dry-run with 5 master nodes in us-east-1 (zones a,b,c,d,e), and save output to kops_cluster.yaml
2. kops create -f kops_cluster.yaml
3. kops replace -f kops_cluster.yaml
4. kops update cluster ABC --out=./ --target=terraform
5. terraform apply

Change cluster zones:

1. edit for 3 zones (a,b,c) in kops_cluster.yaml (rename IG d and e to a-2 and b-2)
2. kops replace -f kops_cluster.yaml --force
3. kops update cluster ABC --out=./ --target=terraform
4. terraform apply

5. What happened after the commands executed?

The terraform apply gets stuck waiting for the etcd volumes to get destroyed.
I tried manually terminating the EC2 instances using these volumes.
This allowe a second try of change steps 2-4 to succeed but kops validate no longer validates so not possible to do a kops rolling-update.

6. What did you expect to happen?

I expected the terraform apply to complete successfully and then the rolling-update to be possible and successful.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

[olemarkus]

/assign

[olemarkus]

Can you have a look at the linked PR and see if this works for you?

[schollii]

How do I test this, do I just kops edit cluster, do I kops delete ig d and e

[olemarkus]

If you can build a kops client from that PR and just do this how you expect to do this, it would be good. You can ping me on slack if you need any direct assistance. Hopefully the validation logic should guide you.

I'll stress again that when it comes to deleting master IGs you need to take care yourself not to break quorum (i.e you can only delete less than half of the master IGs at the same time). If you break quorum you may have to restore etcd from backup.