e2e: manifest url cache

[pohly]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

Manually mirroring files from upstream URLs is error prone. It is hard
to verify during a review whether the files are actually in sync with
their origin, or even determine what that origin is.

On the other hand, directly downloading via URLs during a test or
build increases the risk of random failures in CI jobs.

This problem gets addressed by:

• referencing test files via URL
• registering those URLs in the test/e2e/e2e.test binary
  before tests run
• loading the content of the files from a local cache
  (either via file or bindata)
• adding tooling that updates resp. verifies the cached files

The URL cache is placed under test/e2e/testing-manifests even though
the files might not always be .yaml manifest files because that
directory already gets built into the binary via bindata.

The hack/verify-url-cache.sh is meant to be used in a (to be created)
Prow pre-submit job that triggers only when test/e2e gets
updated. That way the normal "verify" job will not be affected by
the (potentially flaky) download verification.

Registering normal files is also beneficial, because then the test
suite can check in advance whether the file is available. Otherwise a
complete test run is needed to detect a missing file, or the error
might not be detected at all when the test that uses it doesn't run.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pohly
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: gmarek

If they are not already assigned, you can assign the PR to them by writing /assign @gmarek in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• hack/OWNERS
• test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pohly]

/hold

kubernetes-retired/drivers#124 and #71667 need to be merged first.

[pohly]

/cc @msau42

This is the proposal for referencing manifest in the upstream repos. Does this concept look right to you?

[pohly]

/hold cancel

Although kubernetes-retired/drivers#124 was merged, we can't use the YAML files from that repo because they reference the wrong images. I changed the PR such that it only references those files via URL that are usable, i.e. the sidecar RBAC rules.

While working on the update it occurred to me that registering plain files is useful to detect missing files quickly and reliably, so I've added that.

[pohly]

/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-e2e-gce

[pohly]

The feedback from SIG-testing in the meeting today was that these .yaml files shouldn't be needed by the E2E test suite at all. Instead, drivers should be installed as part of the test cluster setup. How that could work is open though, and I suspect that we'd still need to move .yaml files around.

As a result, I'm not sure whether I can get someone from SIG-testing to review this PR. They seem more interested in that long-term solution.

[k8s-ci-robot]

@pohly: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pohly]

The long-term direction is to move the driver installation into the test cluster setup phase. Therefore the approach with framework.testfiles is at best an interim solution. Therefore no-one was inerested enough to review and approve it and I am closing the PR.

SIG-storage will have to continue with manually keeping .yaml files in sync. Once it is clear where those files will be used in the long run (might still be in kubernetes/kubernetes, just in a different directory), some automatism for updating and verifying them can be introduced.