testing: testing with -json option is not debuggable

[zolotov]

Continuation of #22789 and relates to #22996

Suggested solution doesn't allow debug tests and use JSON output simultaneously.

[bradfitz]

I don't understand.

What do you mean by debuggable?

[zolotov]

Literally, I cannot debug a test with Delve.

[ianlancetaylor]

Sorry, can you spell out in more detail what you want to do?

Do you want to run Delve on a test, while looking at the JSON output from the test? Why is that an important use case?

[zolotov]

Yes, I want to debug my tests and have JSON output at the same time, not sure what I can add to this.

It's important because otherwise IDE or any other tool that provides debug support for tests won't be able to use brand new JSON output. Particularly I won't be able to fix a lot of issues that users asked us to fix in testing.

[ianlancetaylor]

Your IDE is going to be parsing the output of the program you are debugging? I'll just take your word for it. I can't picture that at all.

In that case, why not have your IDE pipe the test output through go tool test2json?

[zolotov]

Your IDE is going to be parsing the output of the program you are debugging?

Yes, it's not different in any way from the current situation when we have to parse messages like ===RUN.

In that case, why not have your IDE pipe the test output through go tool test2json?

Because of #22789, because it's much more complicated and because the issue not only about the IDE: it might be tools for remote debugging on continuous integrations servers or just for a user who wants to see json reports while debugging from the CLI.

[zolotov]

Is this still waiting for info?

[ianlancetaylor]

Well, yes, this is still waiting for info. Can you say precisely what change you want?

If the change you want is for tests to be able to generate JSON output without going through the test2json program, that is not going to happen for the reasons given over at #22996.

[zolotov]

Well, yes, this is still waiting for info. Can you say precisely what change you want?

I'm not asking for particular changes, I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

that is not going to happen for the reasons given over at #22996.

Having a test-binary that reports JSON would fix the issue automatically, but having such binary is not the goal, this is why it's different from #22996

[ianlancetaylor]

I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

I'm sorry for asking the same question over and over again, but I honestly and genuinely do not understand what you mean. I can debug a test by running delve. Whether the test output JSON or not is entirely irrelevant to my debugging experience.

Let me try this a different way. Can you explain, in as much detail as possible, precisely what you do, what you expect to happen, and precisely what happens instead? Thanks.

[zolotov]

I can debug a test by running delve

This is what I want to know. How do you run it?

This is what I can imagine about running Delve and having JSON output:

• dlv <binary> doesn't work because, as we know from cmd/test2json: support emitting JSON directly from the test binary #22996, a testing binary cannot produce JSON output.

• dlv test -json <etc> doesn't work because dlv test is a shortcut for making testing binary and then running dlv <binary>.

• dlv ./foo.test -test.v 2>&1 | go tool test2json is not a cross-platform solution (cmd/test2json: handle running command and processing output #22789). Also, it converts Delve's debug output, which is not desirable.

• dlv go tool test2json ./foo.test -test.v produces JSON output but it doesn't stop at breakpoints, likely it tries to debug the wrong process

[zolotov]

go tool test2json dlv <binary> should work but it converts Delve's output

[ianlancetaylor]

When I'm running Delve, I don't really care whether the test is generating JSON output or not. I agree that it is hard to run Delve and simultaneously pipe the test program output, and only the test program output, into test2json.

Why do you want to do that?

It sounds like maybe you want something along the lines of gdb's MI support (https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html), which would permit this. I don't know whether Delve supports anything like that.

[zolotov]

When I'm running Delve, I don't really care whether the test is generating JSON output or not.

We care because we use Delve right now for debugging and we're trying to move to new JSON testing output in order to fix a bunch of different issues with tests output interpretation.

Delve has a protocol on top of tcp to chat with clients and we use this protocol. Still, Delve prints some debug information to stdout and it would be good not to mix it with tests output.

[ianlancetaylor]

No matter what, it is going to be necessary to separate Delve output from test output. So that sounds like something that needs to be changed in Delve, one way or another.

[zolotov]

Sorry, I don't understand. Could you please elaborate what kind of change in Delve you mean?

In general, I think it's not what third-party tools should worry about. Delve has nothing to do with tests or testing output. The only thing that it's responsible for is debugging some executable binary. The fact that its output will be piped to another tool is out of the scope of its interest.

The output format of a testing tool is the responsibility of the testing tool unless the format is configurable by clients like it's done in some java/ruby/js testing tools.

[cespare]

Still, Delve prints some debug information to stdout and it would be good not to mix it with tests output.

Suppose the test binary could emit JSON. Wouldn't that be interleaved with with this "debug information"?

[zolotov]

@cespare it will, for sure, but I'll be able to distinguish JSON and non-JSON output. Or even complete JSON-test event and any other output.

[ianlancetaylor]

You said "Delve prints some debug information to stdout and it would be good not to mix it with tests output." The only possible way to reliably not mix that up is to modify Delve. You need to be able to distinguish Delve output from the output of the program that Delve is running.

[zolotov]

is to modify Delve

Sorry, still unclear what the change you have in mind. What's the approach you suggest considering that only go tool test2json dlv <binary> may potentially work right now?

[zolotov]

And as I said your suggestion means that every tool that could work with test binaries and JSON output will have to do some changes. It is not scalable approach and it quite weird to require tools make hacks for tests.

I think we'll add profiling support in IDE in future and we'll do profiling tests. The suggested approach means that we'll face the very same problem again.

[ianlancetaylor]

I'm sorry, we are somehow failing to communicate. I still don't understand your problem, and you don't understand what I am saying.

As far as I can tell, you never answered my question above: "Can you explain, in as much detail as possible, precisely what you do, what you expect to happen, and precisely what happens instead?"

Let me also try a different question: what, precisely, would you like to change in Go?

[zolotov]

Hm, I believe I answered the first question: I want to know a way to debug tests with JSON output without messing the output of debugger.

As for the second question about changes in Go, I think if I knew what precisely should be done, I'd made a PR :)

For now, I just declare the issue I faced and would like to have a solution for it.
#22996 might be a possible solution, so this is maybe considered as one more usecase for implementing it. If you have other ideas how to handle this – it's great.

By the way, I just tried to run something similar to go tool test2json dlv <binary>, it works almost fine but it finishes with 130 exit code. Do you know the reasons for such behavior of test2json tool?

That's the command I ran: /Users/zolotov/dev/go/bin/go tool test2json -t /path/to/dlv --listen=localhost:52757 --headless=true --api-version=2 --backend=default exec /path/to/compiled_test -test.v -test.run ^TestCounterClear$

[ianlancetaylor]

Closing because I do not understand anything we can change in the Go distribution to fix this.