Add terraform testing

[rifelpet]

This adds a new verify-terraform make target and hack script that uses the official terraform docker image to validate the .tf files in our integration tests.

This can catch issues like:

• missing required resource properties
• typos in resource property names
• invalid values for resource properties

Unfortunately the test fails unless all referenced external files are present, which means we need to check in the userdata, IAM policies, and SSH public key files used by the terraform code's file() references. I decided to update the regular integration tests to also validate their contents. hack/updated-expected.sh will now also create these files.

Marking WIP because some clusters didn't have all their files created, i think it might be an issue with the expected list of files vs actual list of files. also need to get this connected to a prow job.

Feel free to review the hack script now though. I chose to use docker in order to make it work across platforms more easily. I tried using bazel but only found a terraform rules repo that is two years old so I was a bit hesitant to use it. shelling out from bazel seemed messy too.
ref: #8648

[rifelpet]

I think this is ready for review/merge.

There are failures with google's terraform setup. The hardcoded provider version requires terraform >= 0.12 but the remainder of the syntax requires terraform <0.12. This will be fixed as 0.12 support is added in a future PR.

#8744 will also "conflict" with this PR in the sense that tests will fail if both are merged without one of them requiring changes (I have no preference which lands first, I'll fix the other). This is because this PR adds empty bastion userdata files in order to pass the integration tests, but #8744 will assert that they don't exist.

Because of the failures, I'm making the script non-blocking: it exits 0 even in the case of failures. I'd rather get this merged early to avoid further issues with other PRs and the new data files now being checked in.

Once #8744 is merged and we either fix the google provider or rollout the 0.12 upgrade we can make the script blocking.

[rifelpet]

@mikesplain you mentioned in office hours yesterday that you use Terraform. Any chance you can take a look at this?

[mikesplain]

Absolutely, I’ll take a look when I’m back at my machine.

[mikesplain]

This looks awesome. Thanks for your hard work on this, definitely lots of value here.

Lets get this in.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikesplain, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mikesplain,rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[rifelpet]

I had to add empty userdata files for bastion launch configs (and launch templates) because we still compare them and ensure they exist. I actually have #8744 open which will negate that, but that requires #8737 which is ready for review/merge whenever.

Lastly theres an issue with our golden compare and line endings. the comparison strips carriage returns before comparing:

kops/pkg/testutils/golden/compare.go

Lines 43 to 44 in 90af3a1

	//on windows, with git set to autocrlf, the reference files on disk have windows line endings
	expected = strings.Replace(expected, "\r\n", "\n", -1)

but the additionalUserData field actually adds carriage returns:

kops/pkg/model/resources/nodeup.go

Lines 197 to 198 in 90af3a1

	writer.Write([]byte(fmt.Sprintf("Content-Type: multipart/mixed; boundary=\"%s\"\r\n", boundary)))
	writer.Write([]byte("MIME-Version: 1.0\r\n\r\n"))

meaning that ./hack/updated-expected.sh wont update the file properly and the subsequent test will continue to fail. I'm still working on how to handle this, so i expect my most recent push to fail some tests.

In the mean time, #8737 is ready to go. If it causes merge conflicts with this or #8744 I'll rebase them asap while I try to work through this newline issue.

[hakman]

/lgtm