Render managed files with Terraform

[johngmyers]

This is an attempt to have the ManagedFile task render through Terraform instead of directly to the state store. This would make them be updated at terraform apply time, not during (dryrun) kops update cluster.

This is an attempt to move forward with #9229. A followup PR would make the completed cluster spec a ManagedFile.

Some issues (beyond the fact this is my first time dealing with Terraform):

• I believe GCE also works with Terraform, so this would probably need to be implemented against GSPath as well.
• The integration tests use Terraform with MemFSPath. I'm not quite sure what the code should do in this case.

[rifelpet]

Could we update the integration test manifests to use s3:// instead of memfs:// ? Some of them test cloudformation as well which doesn't have an s3 object resource, so we may need to create separate manifests or test cases for cloudformation vs terraform.

[johngmyers]

It doesn't look like the integration tests have a way to mock the S3Context currently. So that would be an effort in itself.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-ci-robot]

@johngmyers: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kops-verify-packages	a888106	link	/test pull-kops-verify-packages

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[johngmyers]

It's time for this to go in. Currently node boot breaks between the time one does a kops update cluster --target terraform and when instances are created using the new spec. This closes the gap to between the time one does a terraform apply and when instances are created using the new spec.

To cut down on churn we might want to stop having golden files for the kops-version.txt files and many of the addon manifests. But that can be in a subsequent PR.

[rifelpet]

What is the status on the per-LaunchTemplateVersion files idea? I ask because managing per-LTV files in terraform will be non-trivial given terraform's inability to orphan s3 objects in its normal terraform plan; terraform apply workflow.

[johngmyers]

I believe per-LT files is dependent on this. Writing that code will be easier if both non-Terraform and Terraform have the per-LT files cleaned up through reconciliation.

Such code will have to find the existing ManagedFiles, determine which should be kept, and reconcile the others to deleted.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

I was looking into adding support for GCS objects but given that the integration tests use memfs which use an s3 object definition, it will be difficult to add integration test coverage for GCS. Any suggestions on how to achieve that?

[johngmyers]

Since the integration tests use memfs, there's no integration test coverage for S3 either. I don't think that's a problem. What is needed is unit tests for gsfs.