New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow test binaries to identify as uncacheable #23799

Open
dsymonds opened this Issue Feb 13, 2018 · 11 comments

Comments

Projects
None yet
6 participants
@dsymonds
Member

dsymonds commented Feb 13, 2018

The new test caching stuff is neat, except when a test has an external dependency (e.g. it is testing code that hits a web service), and we don't want the test's result to be cached (so that we're always exercising the code against the real world).

There are ways to disable the test caching from the user's perspective (e.g. passing -count=1), but not from the test itself from what I can tell. It'd be nice if tests in this position could do something to indicate to the go tool that its result and output should not be cached.

Some ideas:

  • Have a method on *testing.T that can be invoked to signal this.
  • Have a specified file to touch (e.g. $GOCACHE/something).
  • Have a naming convention for such tests (e.g. must have an External substring).
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 13, 2018

If a test opens a file that is not under GOPATH or GOROOT, then its result will not be cached. Does that seem sufficient?

@dsymonds

This comment has been minimized.

Member

dsymonds commented Feb 13, 2018

As in, I could open /dev/null (OS-specific, alas) to signal that? Is that documented anywhere?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 13, 2018

No, wait, sorry, I got it wrong. If a test opens a file that is under GOPATH or GOROOT, then if that file changes, the cache is ignored. If a test opens a file that is not under GOPATH or GOROOT, that does not affect caching.

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 13, 2018

@skipor

This comment has been minimized.

skipor commented Mar 22, 2018

Similar problem. Testing executable out of a process, by building it in by github.com/onsi/gomega/gexec and run it against golden files. The test result is cached even if executable sources were changed.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Go1.12 Jul 6, 2018

@dsymonds

This comment has been minimized.

Member

dsymonds commented Jul 17, 2018

https://tip.golang.org/doc/go1.11#gocache says this:

Go 1.11 will be the last release to support setting the environment variable GOCACHE=off to disable
the build cache, introduced in Go 1.10. Starting in Go 1.12, the build cache will be required, as a step
toward eliminating $GOPATH/pkg. The module and package loading support described above already
require that the build cache be enabled. If you have disabled the build cache to avoid problems you
encountered, please file an issue to let us know about them.

This is one such issue. It'd be nice if there was something formally supported by Go 1.11 in advance of a major workaround being removed in Go 1.12.

I included a few suggested approaches in my original message. Who's the decider for this?

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 9, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 9, 2018

I'm not convinced this is a property of test cases. The test is rerun if the binary has changed. If the binary has not changed, then I think it is reasonable to say "this binary passed the last time" and leave it out, especially if you've changed one line in one source file and are running go test all or go test ./... to just test everything that line might affect. In that situation, you definitely don't want to run a binary that is completely unchanged just in case the external dependency has changed itself. If we let test cases unconditionally opt out of caching, then that use case is completely gone.

I realize that in some situations you do really want the test to run even if there was a successful run with exactly the same binary in the past. When that's the case, be explicit about it and say -count=1.

Leaving open to look at again in Go 1.12 cycle but very likely the answer is no.

@rsc rsc changed the title from cmd/go: Provide mechanism for a test to self-identify as non-cacheable to cmd/go: allow test binaries to identify as uncacheable Aug 9, 2018

@dsymonds

This comment has been minimized.

Member

dsymonds commented Aug 10, 2018

I don't understand your response. This is only for a small minority of test cases, which would have to opt-in to this. If a test case has said "don't cache my result", then I do want to run that test every time., because that's exactly what the test case has indicated. Now if that's a problem, it's a problem because of the test having external dependencies (or whatever), not because it has opted out of the test caching. The use case of go test ./... is not at all gone; it makes no sense to say that. It's still there. It just runs an extra test that has opted-in to always being run. And that test has presumably done that knowing full well that it may be run even though the test binary doesn't change.

Using -count=1 is a hack. It's user-specific rather than test-specific, and it breaks the use case that you say you want to preserve: namely it'll bust test caching for go test ./... for all tests, not just the one-in-a-hundred (or whatever) test that knows it needs to be uncached.

I see this analogous to t.Parallel that permits a test to signal that there's something special about the test. It wouldn't be used very often, but it carries important information about the test.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 25, 2018

I think for people working on dependencies down in the leaves it is critical that running go test all twice does nothing the second time. If you are the author of a supposedly-uncacheable test, then yes you want it to run anew every time you run it. But other people run your tests too, and my point is that it is not appropriate to force your preference into those use cases.

It would be OK to have a 'mark this as depending on external things' as long as that result was still cached by default, and then we could also add a 'rerun all the tests that depend on external things' as an explicit flag. Not sure what that would look like exactly.

Leaving for Go 1.13.

@rsc rsc modified the milestones: Go1.12, Go1.13 Oct 25, 2018

@josharian

This comment has been minimized.

Contributor

josharian commented Oct 25, 2018

we could also add a 'rerun all the tests that depend on external things' as an explicit flag

Or cmd/go could automatically re-run such tests in the current module, but not in dependencies.

@bcmills bcmills added the GoCommand label Nov 14, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 14, 2018

Or cmd/go could automatically re-run such tests in the current module, but not in dependencies.

Or, more uniformly, it could re-run tests that matched patterns (foo/...) but not meta-packages (all). But one of the explicit examples that @rsc gave was ./..., which is in the current module.

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 14, 2018

As a workaround, I suppose you could always get the intended effect by using a //go:generate directive to change a random number in a file:

package external

//go:generate bash -c 'echo -e "package external\nconst S = \"$(head -c 8 /dev/random | base64)\"" > random.go'
package foo_test

import _ "example.com/external"
go generate external && go test ./...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment