testing: using "go tool test2json dlv test-binary" marks the last test as failed on Mac OS

[nd]

This is continuation of #23033 which is still actual in go1.13.

To reproduce the issue create <GOPATH>/src/debugTest/foo_test.go:

package foo

import "testing"

func Test1(t *testing.T) {
}

compile it:

go test -c -o testBinary -gcflags "all=-N -l" debugTest

run:

go tool test2json dlv --listen=:2345 --headless exec testBinary -- -test.v

and connect debugger from another console: dlv connect :2345.

In the Delve console type continue and once the testBinary terminates, exit the delve client via the exit command.

Test2json produces output:

{"Action":"run","Test":"Test1"}
{"Action":"output","Test":"Test1","Output":"=== RUN   Test1\r\n"}
{"Action":"output","Test":"Test1","Output":"--- PASS: Test1 (0.00s)\r\n"}
{"Action":"output","Test":"Test1","Output":"PASS\r\n"}
{"Action":"fail","Test":"Test1"}

This happens because on Mac OS Delve runs programs via debugserver and debugserver uses \r\n line-endings. For most of test output it is not a problem because test2json check prefixes and is not affected by line-ending differences. But for final PASS or FAIL the exact Equal check is used and it fails due to added \r.

This behavior was accidentally fixed in d47526e, but judging by comments in #31969 the fix will be reverted later.

I understand that this is not a bug in test2json. Test2json assumes the output is produced by the test framework and the test framework always uses \n. It is possible to wrap Delve and replace PASS\r\n in its output to PASS\n, but every test2json client will have to do that. Maybe it is possible to support this case in test2json itself?

In #23033 there was a question: why one would want to debug a test and also see the its final status. The most relevant use-case I can think of is a modification of a program state during debug session. Let's say I have a complex algorithm test for which started to fail. During a debug session I've found a potential problem. I cannot easily change the sources and rerun a test because the fix is complicated, but I can change the program state at runtime. I want to verify that changing some value in a program produces a desirable effect and test starts to pass. At the moment it is not possible as the test status is always "fail".

[ianlancetaylor]

Why do you run

go tool test2json dlv --listen=:2345 --headless exec testBinary -- -test.v

instead of

dlv --listen=:2345 --headless exec testBinary -- -test.v

? How does running under go tool test2json make it easier to debug the program?

[ianlancetaylor]

Why does delve output \r\n rather than \n when its standard output is not a terminal? Where is that coming from?

[nd]

Test2json is used to simplify test output parsing, json is easier to parse than the raw output produced by test executable. Test2json output is consumed by an editor and is shown in a dedicated UI where each test has an icon according to its status. It is easier to grasp tests status from such a view than from reading output of all tests. Test2json doesn't change the debugging experience per se, but it is useful to be able both debug a test and see its outcome once debug session is over.

If I understand correctly Delve redirects debugserver output to its stdout without changing anything. @aarzilli could you please confirm?

[aarzilli]

This thing with tests getting \r\n on macOS came out before and I couldn't figure out where it happens. The way it works is that delve runs the test executable under debugserver and then debugserver sends us the output in a stop packet (it's the 'O' kind). It already contains the \r there so either debugserver is adding it (werid) or possibly the tty layer in the kernel (weirder).

[asilverman]

@rsc Why not simply add :
|| bytes.Equal(line, []byte("PASS\r\n"))
to line 158 and 161 and be done with it ?

go/src/cmd/internal/test2json/test2json.go

Lines 156 to 167 in 5c6f427

	func (c *converter) handleInputLine(line []byte) {
	// Final PASS or FAIL.
	if bytes.Equal(line, bigPass) || bytes.Equal(line, bigFail) {
	c.flushReport(0)
	c.output.write(line)
	if bytes.Equal(line, bigPass) {
	c.result = "pass"
	} else {
	c.result = "fail"
	}
	return
	}

[peterhuson]

bump
This breaks GoLand test parsing: https://youtrack.jetbrains.com/issue/GO-6493

[nd]

It looks like debugserver creates a new pseudo terminal for the inferior and by default terminal has the ONLCR flag set which maps NL to CR-NL.

[gopherbot]

Change https://go.dev/cl/428715 mentions this issue: test2json: tolerate \r\n as a newline separator in test output

[gopherbot]

Change https://go.dev/cl/443596 mentions this issue: testing: fix many test2json inaccuracies