kubetest2 - Implement create/validate/delete cluster functionality

[rifelpet]

This connects the kubetest2-kops --up --down flags to run kops create cluster kops validate cluster and kops delete cluster.

I added a preliminary set of flags that we'll pass to kops create cluster but there are many more flags we'll add to the Deployer struct (zones, instance counts and sizes, etc.).

New flags:

$ cd tests/e2e; go run kubetest2-kops/main.go --help
Usage:
  kubetest2 kops [Flags] [DeployerFlags] -- [TesterArgs]

Flags:
      --artifacts string   top-level directory to put artifacts under for each kubetest2 run, defaulting to "${ARTIFACTS:-./_artifacts}". If using the ginkgo tester, this must be an absolute path. (default "/Users/prifel/work/src/k8s.io/kops/tests/e2e/_artifacts")
      --build              build kubernetes
      --down               tear down the test cluster
  -h, --help               display help
      --run-id string      unique identifier for a kubetest2 run (default "b87ccf2b-990b-4f66-916c-7680c12ee089")
      --test string        test type to run, if unset no tests will run
      --up                 provision the test cluster

DeployerFlags(kops):
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --alsologtostderr                  log to standard error as well as files
      --cloud-provider string            Which cloud provider to use
      --cluster-name string              The FQDN to use for the cluster name
      --env strings                      Additional env vars to set for kops commands in NAME=VALUE format (default [])
      --kops-binary-path string          The path to kops executable used for testing
      --kops-root string                 Path to root of the kops repo. Used with --build.
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --ssh-private-key string           The path to the private key used for SSH access to instances
      --ssh-public-key string            The path to the public key passed to the cloud provider
      --ssh-user strings                 The SSH users to use for SSH access to instances (default [])
      --stage-location string            Storage location for kops artifacts. Only gs:// paths are supported.
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Example:

$ cd tests/e2e;  go run kubetest2-kops/main.go --up --down --cluster-name foo.k8s.local --cloud-provider aws --kops-binary-path $(which kops)
I1022 20:58:22.589790   34345 app.go:87] ID for this run: "0d17c334-838d-47be-9bf2-20cb5e4e8762"
I1022 20:58:22.589960   34345 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip
2020/10/22 20:58:22 failed to get external ip from metadata service: Get "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip": dial tcp: lookup metadata.google.internal: no such host
I1022 20:58:22.592784   34345 http.go:37] curl https://ip.jsb.workers.dev
I1022 20:58:22.909951   34345 up.go:53] /path/to/kops create cluster --name foo.k8s.local --admin-access 1.2.3.4/32 --cloud aws --master-count 1 --master-size c5.large --master-volume-size 48 --node-count 4 --node-volume-size 48 --override cluster.spec.nodePortAccess=0.0.0.0/0 --ssh-public-key  --zones eu-west-2a --yes
I1022 20:58:24.011720   34347 featureflag.go:158] FeatureFlag "SpecOverrideFlag"=true
...

I'll set up an optional presubmit prow job for this once i get the boskos client setup which I'll do next.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

This is copied from test-infra/scenarios/kubernetes_e2e.py with the following simplifications:

• Batch jobs (triggered by tide when merging a batch of PRs) included "batch" in the name but it was always hashed with the unique BUILD_ID, so it was easier to remove "batch" and just hash the BUILD_ID which is just as unique
• Presubmits used the PR number if the job was configured to tear down the previous cluster, but none of the kops jobs do that, so now the cluster names of all jobs will only depend on JOB_NAME and BUILD_ID

[rifelpet]

This is mostly copied from kubetest/kops.go along with http.go above

[olemarkus]

Looks generally good to me.

There is a check that ensures we are targetting aws in there, and then later on there are things that also relates to gcp, but I suppose that comes from copying over existing code. It may be simpler if the code is aws-only for now, but it is no big deal if we just leave it in either.

[rifelpet]

You're right, I copied some gce values as I was porting over code. There's also some unused fields (ssh private key) that I copied since they're defined alongside ssh public key. Im happy to back those out if it reduces confusion.

[rifelpet]

/retest

[olemarkus]

Nah. Fine for me.
/lgtm