Put versioned API of cluster into state store

[johngmyers]

This addresses two problems:

1. The "completed" cluster spec, stored in the file cluster.spec in the state store, is stored using the unversioned API. This is contrary to apimachinery rules, which state that APIs should be persisted in a versioned form of the API, and limits our ability to change the unversioned API.

2. The cluster.spec file is written at kops edit (etc.) time, causing some aspects of the cluster's operation to change prior to running kops update cluster --yes.

This PR starts writing the versioned API to a new cluster-completed.spec. This new file is only written at kops update cluster --yes time.

This PR fixes readers of the completed cluster spec to deserialize as the versioned API.

It also does not address storage of cluster-completed.spec through a k8s state store.

I am considering a future PR that would write the cluster-completed.spec file as a ManagedFile, replacing the VFS API method added here.

[johngmyers]

/kind bug

[johngmyers]

This will have some effect on Terraform users, who currently expect to run kops update cluster without --yes. Unless we can get Terraform to update the state store we'll need to update the instructions.

[johngmyers]

My plan to address the issue of the instancegroup specs being written prior to kops update cluster --yes, at least as far as nodeup, is to remove the InstanceGroup field from NodeupModelContext, moving all necessary information to the NodeupConfig.

[johngmyers]

I'm putting a hold on this due to the Terraform issue. For now I'll pursue moving state into the NodeupConfig.

[rifelpet]

Unless we can get Terraform to update the state store we'll need to update the instructions.

Would managing the cluster and instance group manifests via terraform fix the issue? TF does have the aws_s3_bucket_object resource type. Rather than kops uploading the object to s3 directly it could write the object to the .tf files and have terraform manage them. Would that be sufficient? We wouldn't use it for any of the PKI secrets, just the manifests

[johngmyers]

@rifelpet that would probably address the issue, modulo race conditions during the update itself.

[johngmyers]

/test pull-kops-e2e-kubernetes-do-kubetest2

[johngmyers]

/cc @justinsb

[johngmyers]

/test pull-kops-e2e-k8s-gce

[olemarkus]

Still looks good to me

/lgtm

[k8s-ci-robot]

@johngmyers: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kops-verify-cloudformation	917a16e	link	/test pull-kops-verify-cloudformation
pull-kops-verify-hashes	917a16e	link	/test pull-kops-verify-hashes
pull-kops-e2e-k8s-containerd	917a16e	link	/test pull-kops-e2e-k8s-containerd
pull-kops-e2e-kubernetes-do-kubetest2	30faa3a	link	/test pull-kops-e2e-kubernetes-do-kubetest2
pull-kops-e2e-k8s-gce	30faa3a	link	/test pull-kops-e2e-k8s-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[johngmyers]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [olemarkus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment