New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: support compiling all tests without running #15513

Open
bradfitz opened this Issue May 3, 2016 · 26 comments

Comments

Projects
None yet
@bradfitz
Member

bradfitz commented May 3, 2016

I want this to work:

$ go test -c std cmd
cannot use -c flag with multiple packages

But I don't actually care about the binaries. I just want to test that all the tests can compile, and how fast. @randall77 also wants this for SSA coverage reasons.

@bradfitz bradfitz added this to the Unplanned milestone May 3, 2016

@cespare

This comment has been minimized.

Contributor

cespare commented May 3, 2016

Is go test -run xxxxx std cmd a reasonable workaround for at least some of those use cases?

@josharian

This comment has been minimized.

Contributor

josharian commented May 3, 2016

And I want this, but I do want the test binary. I'd use it to compile tests, change the compiler, compile tests again, and then run before/after benchmarks. (All in a script, of course.) Like toolstash for std tests. I currently do something similar but with go list std and a lot of bookkeeping and go test -c invocations.

It seems to me we could generate a testmain that invokes all the tests in all the included packages. There are some questions: How do you know from the -v output which package this particular instance of TestReader is from? How does the -run filter work? And so on. My hand-waving answer is: Mimic the UI for subtests and sub benchmarks, which already have this nesting problem; the nesting is just up a level rather than down.

@minux

This comment has been minimized.

Member

minux commented May 3, 2016

@josharian

This comment has been minimized.

Contributor

josharian commented May 3, 2016

What if we leave that up to the user to diagnose? If package A's test interferes with package B's use of package A, then don't test them together. This sort of issue can already arise in a single package.

@minux

This comment has been minimized.

Member

minux commented May 3, 2016

@josharian

This comment has been minimized.

Contributor

josharian commented May 3, 2016

They'll know when the test fails. If the tests pass when run independently (i.e. go test ./..., no -c), then there's clearly an interaction problem.

@minux

This comment has been minimized.

Member

minux commented May 3, 2016

@josharian

This comment has been minimized.

Contributor

josharian commented May 4, 2016

I can't imagine someone wanting to run go test -c ./... on their entire GOPATH unless they had a very good reason. I can, however, see this for a particular project or set of packages, in which case this becomes very useful. Think e.g. of cross-compiling all the tests for your project and having to only scp a single binary.

cmd/go doesn't support negative patterns

Yes, this is unfortunate.

I apologize, but I don't quite understand the resistance here. It seems like the resistance is "something could go wrong". But this doesn't really seem like a big footgun to me. And when this is helpful, like cross-compiling all tests for a big project or keeping around an old version for benchmarking or testing compilation speed, it is very helpful, and users will be motivated to make it work (or not use it). And it is entirely backwards compatible.

@minux

This comment has been minimized.

Member

minux commented May 4, 2016

I think this issue gives one reason for do go test -c ./... for the
whole GOPATH: test the SSA compiler code generation.

The key point is most tests are not prepared for this.
How many packages support go test -count=2?
Even some of the std packages don't, until they're
fixed not long ago.

And how do you propose to handle conflicting flags added
by tests and multiple TestMains? Again, these features
are just not designed to combine tests for multiple packages
into a single executable.

Also note that tests for a package needs to run at the
directory for the package, how to handle that when
tests for multiple packages are combined into one binary?

(FTR, I have been thinking about this feature a long time
ago, but my conclusion was and still is that the whole thing
is not designed for this.)

@josharian

This comment has been minimized.

Contributor

josharian commented May 4, 2016

How many packages support go test -count=2? Even some of the std packages don't, until they're
fixed not long ago.

Exactly. And they got fixed because people cared. Same situation here, I'd say.

conflicting flags added by tests

Sorry, what do you mean by this?

multiple TestMains

Refuse to compile, with a lucid error message. Rare.

tests for a package needs to run at the directory for the package

os.Chdir? Note that the go command has the directory available at compilation time. This is a bit ugly for the cross-compilation case, but we could print a warning instead of failing when os.Chdir fails.

I have been thinking about this feature a long time
ago

Yeah, I recall there being another issue about this but couldn't find it.

I agree that there are limitations, I just see them as limitations rather than showstoppers.

@minux

This comment has been minimized.

Member

minux commented May 4, 2016

By conflicting flags added by tests, consider two
packages that both add -update flags to update
golden output of their test data.

The code will compile, but will fail at runtime with
a flag redefined panic.

There are just too many global states, hence I
said the whole system is not designed for this.

@mattklein123

This comment has been minimized.

mattklein123 commented May 26, 2016

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz

@davecheney

This comment has been minimized.

Contributor

davecheney commented May 26, 2016

If you want to compile but run not tests,

 go test -run=nope ./...

Replace nope with another pattern if you have test cases that match.

On Fri, 27 May 2016, 03:45 Matt Klein notifications@github.com wrote:

I'm new to Go, but FWIW, the lack of this feature is one of the most
frustrating things that I have found so far about the tooling. I tend to
write production code and test code at the same time, and during my normal
dev cycle I want to see if things compile many more times than I want to
run the tests. So right now I'm doing roughly the same thing as stated
above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and
have it be up to the user to deal with conflicts as suggested by @bradfitz
https://github.com/bradfitz


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15513 (comment)

@mattklein123

This comment has been minimized.

mattklein123 commented May 26, 2016

@davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.

@davecheney

This comment has been minimized.

Contributor

davecheney commented May 27, 2016

Please try -run=nope

On Fri, 27 May 2016, 09:56 Matt Klein notifications@github.com wrote:

@davecheney https://github.com/davecheney I'm probably doing something
stupid but running that command still causes all the tests to execute. I
tried:

go test -run nope ./...

Also still runs tests.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15513 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8xNrwQtBDQEF3AeZEF-377nOLMUks5qFjMwgaJpZM4IV2e-
.

@mattklein123

This comment has been minimized.

mattklein123 commented May 27, 2016

I tried both:

mklein@vm:~/Source/go/src/github.com/<redacted>$ go test -run=nope ./...
ok      github.com/<redacted>   0.025s
@cespare

This comment has been minimized.

Contributor

cespare commented May 27, 2016

-run=nope and -run nope are equivalent.

@mattklein123 no tests are being run (probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

@davecheney

This comment has been minimized.

Contributor

davecheney commented May 27, 2016

That is correct. The test binary is built and executed, but none of the
test cases will be executed because they don't match the -run filter

On Fri, May 27, 2016 at 10:11 AM, Caleb Spare notifications@github.com
wrote:

-run=nope and -run nope are equivalent.

@mattklein123 https://github.com/mattklein123 no tests are being run
(probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15513 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8u95PM2MDjAGCytHBIVmp0py0RIks5qFjaxgaJpZM4IV2e-
.

@juRiii

This comment has been minimized.

juRiii commented Sep 2, 2016

i prefer -run ^$

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 8, 2016

My motivation is for the builders, and test sharding.

The problem with go test -run=^$ std cmd is that it deletes the *.test binaries afterwards. I want the test binaries, to ship around the network and shard out isolated test execution. But I want to burn CPU as hard as possible where it's not latency sensitive. (That is, I can run compilation jobs at 100% CPU, but tests are often flaky at 100% if they sleep or depend on time.)

@bradfitz bradfitz modified the milestones: Go1.8Maybe, Unplanned Sep 8, 2016

@bradfitz bradfitz changed the title from cmd/go: support compiling all tests without running to proposal: cmd/go: support compiling all tests without running Sep 8, 2016

@minux

This comment has been minimized.

Member

minux commented Sep 9, 2016

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 9, 2016

I can do it myself, but my concern is that I can't keep the CPU busy as well as the cmd/go binary could.

cmd/go can load the dependency graph once, come up with a plan, and keep a certain number of worker processes running at all times.

Any coordination I do trying to run multiple cmd/go binaries wouldn't be as good.

Definitely low priority, though.

@rsc rsc modified the milestones: Go1.9, Go1.8Maybe Oct 20, 2016

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 19, 2016

There are at least three different things people might want here (and have asked for either here or other places):

  • Build all binaries and don't save them.
  • Build all binaries and do save them (maybe in a directory tree).
  • Compile but don't link (for quick turnaround in save hooks).

I propose we keep accumulating these kinds of go command issues and try to take a coherent look at them in the first half of next year.

I'll create a new GoCommand label. Feel free to label other (proposal or non-proposal) issues with that too.

Thanks.

@gopherbot

This comment has been minimized.

gopherbot commented May 3, 2017

CL https://golang.org/cl/42531 mentions this issue.

@bradfitz bradfitz modified the milestones: Unreleased, Go1.9 Jul 22, 2017

@FlorianUekermann

This comment has been minimized.

Contributor

FlorianUekermann commented Sep 26, 2017

It would be great if this enabled "buildall.bash" to also check if all tests compile. That would have saved me some embarrassment when submitting CLs.

@shivakumargn

This comment has been minimized.

Contributor

shivakumargn commented Dec 16, 2017

I have a large C based codebase which has Go wrappers. I would like to write Go test packages (only test *.go code) as test drivers for the C code, using the go wrappers, create go test binary and ship the binary with the installed software as a test suite for the environment. This is not a typical test suite in a dev machine.
The structure of go source code and the godoc helps in organizing the test suite nicely out of the box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment