Prefix etcd cluster names with letters

[hakman]

Refs: #9982

/cc @rifelpet

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

this looks good to me though it doesn't fix existing clusters being upgraded to Terraform 0.12.

[hakman]

Not sure if we can have more than a release note for those running Terraform 0.12. At least with this new clusters could be created.

[rifelpet]

what would the release note say? It seems like we should either use a different terraform resource name if the original name starts with a number and have the release notes instruct users to terraform state mv, or walk users through changing the etcd cluster member names which isn't clearly understood or documented, and likely involves control plane downtime.

Users will experience this issue when upgrading existing clusters, so we should be prepared to handle that.

[hakman]

I think this is a start. If someone bumps into this issue with an existing cluster, will be a pain, but I don't see any easy solution. It's not just about renaming resources. If the community will find some way to fix things manually and documents the steps, I would be happy to look at that.

[rifelpet]

i think the easiest solution is to have kops change the terraform resource name that it generates when it would otherwise start with a number in this situation. Then we know that anyone that had their resource name change would have otherwise ran into this issue and they'll need to terraform state mv the resource to its new name. The fix only impacts users that are directly affected by the issue, and the fix is not disruptive to the cluster itself.

[hakman]

That would require more work and testing.

[justinsb]

I find it hard to reason about whether this is safe. Can we first create a test case that hits the numbers problem (in the create_cluster case), and then rebase this on that so we can see the change?

[hakman]

@justinsb I added the test as part of #10365.

[rifelpet]

/lgtm

we will merge this plus #10424 and an env var check as discussed in the referenced GH issue

[hakman]

/retest