proposal: cmd/go: allow test binaries to identify as uncacheable

[dsymonds]

The new test caching stuff is neat, except when a test has an external dependency (e.g. it is testing code that hits a web service), and we don't want the test's result to be cached (so that we're always exercising the code against the real world).

There are ways to disable the test caching from the user's perspective (e.g. passing -count=1), but not from the test itself from what I can tell. It'd be nice if tests in this position could do something to indicate to the go tool that its result and output should not be cached.

Some ideas:

• Have a method on *testing.T that can be invoked to signal this.
• Have a specified file to touch (e.g. $GOCACHE/something).
• Have a naming convention for such tests (e.g. must have an External substring).

[ianlancetaylor]

If a test opens a file that is not under GOPATH or GOROOT, then its result will not be cached. Does that seem sufficient?

[dsymonds]

As in, I could open /dev/null (OS-specific, alas) to signal that? Is that documented anywhere?

[ianlancetaylor]

No, wait, sorry, I got it wrong. If a test opens a file that is under GOPATH or GOROOT, then if that file changes, the cache is ignored. If a test opens a file that is not under GOPATH or GOROOT, that does not affect caching.

[skipor]

Similar problem. Testing executable out of a process, by building it in by github.com/onsi/gomega/gexec and run it against golden files. The test result is cached even if executable sources were changed.

[dsymonds]

https://tip.golang.org/doc/go1.11#gocache says this:

Go 1.11 will be the last release to support setting the environment variable GOCACHE=off to disable
the build cache, introduced in Go 1.10. Starting in Go 1.12, the build cache will be required, as a step
toward eliminating $GOPATH/pkg. The module and package loading support described above already
require that the build cache be enabled. If you have disabled the build cache to avoid problems you
encountered, please file an issue to let us know about them.

This is one such issue. It'd be nice if there was something formally supported by Go 1.11 in advance of a major workaround being removed in Go 1.12.

I included a few suggested approaches in my original message. Who's the decider for this?

[rsc]

I'm not convinced this is a property of test cases. The test is rerun if the binary has changed. If the binary has not changed, then I think it is reasonable to say "this binary passed the last time" and leave it out, especially if you've changed one line in one source file and are running go test all or go test ./... to just test everything that line might affect. In that situation, you definitely don't want to run a binary that is completely unchanged just in case the external dependency has changed itself. If we let test cases unconditionally opt out of caching, then that use case is completely gone.

I realize that in some situations you do really want the test to run even if there was a successful run with exactly the same binary in the past. When that's the case, be explicit about it and say -count=1.

Leaving open to look at again in Go 1.12 cycle but very likely the answer is no.