create make test cached k/k presubmit (optional, always_run: false)

[BenTheElder]

/cc @spiffxp @liggitt @dims

It's time to ship @Katharine's cache work to presubmit :-)

This is the first step, just adding a job we can manually trigger to sanity check.

Once that's done I intend to make it at minimum skip_report: true but always_run: true (and still optional: true).

See how the current CI jobs are faring:
https://prow.k8s.io/?job=ci-kubernetes*make*

Part of kubernetes/enhancements#2420

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [BenTheElder]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

TODO: what timeout do we really want here? This one is observed from GKE's equivalent infra, and is at least lower than the 600s below.
TODO: should KUBE_RACE=-race be default? In bazel we just had race by default IIRC. Ideally locally I should just make test and it should do the preferred thing ... I don't know why it isn't default yet though.

[BenTheElder]

Fun fact:
https://prow.k8s.io/?job=periodic-kubernetes-bazel-test-master ~22+ minutes
https://prow.k8s.io/?job=ci-kubernetes*make* ~6 minutes (for the cached job)

Bazel is basically not caching unit tests still even after disabling the run per test bump ...?

[dims]

/lgtm

yes please!

[k8s-ci-robot]

@BenTheElder: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key make-test.yaml using file config/jobs/kubernetes/sig-testing/make-test.yaml

In response to this:

/cc @spiffxp @liggitt @dims

It's time to ship @Katharine's cache work to presubmit :-)

This is the first step, just adding a job we can manually trigger to sanity check.

Once that's done I intend to make it at minimum skip_report: true but always_run: true (and still optional: true).

See how the current CI jobs are faring:
https://prow.k8s.io/?job=ci-kubernetes*make*

Part of kubernetes/enhancements#2420

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

however first presubmit run was 26m48s, investigating
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/92316/pull-kubernetes-make-test/1364361467876347904

[BenTheElder]

Ah, go test -race disables caching.

The rule for a match in the cache is that the run involves the same
test binary and the flags on the command line come entirely from a
restricted set of 'cacheable' test flags, defined as -cpu, -list,
-parallel, -run, -short, and -v. If a run of go test has any test
or non-test flags outside this set, the result is not cached. To
disable test caching, use any test flag or argument other than the
cacheable flags. The idiomatic way to disable test caching explicitly
is to use -count=1. Tests that open files within the package's source
root (usually $GOPATH) or that consult environment variables only
match future runs in which the files and environment variables are unchanged.
A cached test result is treated as executing in no time at all,
so a successful package test result will be cached and reused
regardless of -timeout setting.

test docs

So we can have very fast presubmits, but without -race. We could run with -race in a periodic ...
Or we can give up on caching and accept that < 30m is OK for kubernetes presubmits for now.

[BenTheElder]

AFAICT we'd have to patch go test to cache with -race:
https://golang.org/src/cmd/go/internal/test/test.go#L1318

[BenTheElder]

It is however still within a few minutes of the bazel job, and passing.

[BenTheElder]

#20991 to disable -race so we can get the results with caching, we can flip it back on if we decide it's more important (these jobs are not blocking / monitored yet except by me).