build: unclear how to rerun test failure

[lexprfuncall]

When a test failure is encountered it is not always obvious how to re-run the test.

It would be useful for the test harness (whatever one is involved) to print a command
line that a devloper could later invoke to reproduce the failure.

[rsc]

Comment 1:

This report does not contain much actionable information.
What kind of failure are you seeing that you don't know how to rerun?
It is true that the test script does not include every command that is being run, but
that's mostly by design. 99% of the time that is noise.
The short answer to the general question is that if the failure looks like:
ok      net/smtp    0.041s
ok      net/textproto   0.027s
ok      net/url 0.022s
ok      os  2.116s
ok      os/exec 1.189s
ok      os/signal   0.454s
ok      os/user 0.025s
ok      path    0.047s
ok      path/filepath   0.958s
runtime: unknown argument frame size for reflect.methodValueCall called from 0x7878c
[reflect_test.func·009]
fatal error: invalid stack
...
FAIL    reflect 0.126s
then that means you should rerun 'go test reflect'. 
If the failure happens at the end and refers to a file like test/fixedbugs/bug123.go you
can cd into the test directory and run 'go run run.go -- fixedbugs/bug123.go'. The --
keeps 'go run' from trying to handle the Go file itself.
For other failures, there should be enough tracing prints (they begin with #) in the
output to follow along in run.bash. If not, we should fix that.

Labels changed: added priority-later, removed priority-triage.

[lexprfuncall]

Comment 2:

There are many different kinds of tests that are run and, as you point out, each kind of
test requires a different invocation on the command line to run it.  Moreoever, running
just the failed test (as opposed to the collection of tests that include that test) is
something that is not entirely obvious and also varying.
While there are plenty of # lines, they are not entirely hepful.  Consider the following
failure
# ../test/bench/go1
ok      _/usr/local/google/home/cshapiro/go/test/bench/go1  2.360s
# ../test
SIGSEGV: segmentation violation
PC=0x41c671
What do I do?  The 28 stack traces that follow do not contain any obvious clues about
what was running when the failure occured.  Any good programmer can, given the time,
eventually reverse engineering what the right test invocation is but that time is better
spent diagnosing the failure.
What I want to see after a failure like that is a command line that I can paste into the
shell so I can re-run just the test that caused the failure.

[rsc]

Comment 3:

I think the main thing that is missing is a document explaining how to interpret the
all.bash output. 
Even if we were going to arrange for these command lines to be printed on every failure,
the crash you reported in comment #2 (at least judging from the part you included) looks
like it is the test runner that has crashed, not any specific test. The sad fact of
self-hosted systems like ours is that if you break the runtime or the compiler there's
very little stable left to stand on. Assuming it is the test runner that has failed, you
would not have gotten a useful command line even if we were trying to print them.
One possibility is that if this is an ongoing problem you could add 'set -x' to the top
of your src/run.bash.

[lexprfuncall]

Comment 4:

The crash I reported turns out to have been caused by a test, and not the test runner.
I think it would be more useful for the all.{bash,bat,rc} output to be self describing.
I believe this is practical.  The test suite used by the Dart team uses a test harness
written in Dart that runs a half-dozen different test suites with thousands of tests. 
Each failure is reported with a command line that can be pasted back into the shell to
repeat the failure.  The turn around time for identifying and fixing a failing test is
very low.

[bradfitz]

Comment 5:

> The crash I reported turns out to have been caused by a test, and not
> the test runner.
I don't believe that.  run.go doesn't produce that output.  That looks like run.go
itself crashed (i.e. the test runner).

[lexprfuncall]

Comment 6:

I'm not sure I am following you, if run.go does not produce that output how does that
indicate that run.go crashed?
For what it's worth, I ended up isolating my crash by sticking a "set -x" at the top of
the shell script, repating the execution until I could identify a crashing test,
compiling the test standalone, and debugging from there.

[bradfitz]

Comment 7:

If a test under run.go crashed, run.go would've reported it differently.
But run.bash does:
(xcd ../test
unset GOMAXPROCS
time go run run.go || exit 1
) || exit $?
The only way I could see to have gotten that output is for go itself to have crashed
(the go running "go run run.go") or the resultant run.go binary to have crashed.
But not a single unit test under test/...

[lexprfuncall]

Comment 8:

I wish we had this conversation eleven days ago so I could have showed you the
environment then and test I eneded up debugging.
Nevertheless, this kind of question should be eliminated if the test system was better
about reporting failures and how to reproduce them.
While we cannot rule out the chance that the harness is failing, experience in other
projects has shown that this is rarely, if ever, a problem.

[rsc]

Comment 9:

This issue was closed by revision 8ce9a4f.

Status changed to Fixed.