Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: testing with -json option is not debuggable #23033

Closed
zolotov opened this issue Dec 7, 2017 · 24 comments

Comments

Projects
None yet
5 participants
@zolotov
Copy link
Contributor

commented Dec 7, 2017

Continuation of #22789 and relates to #22996

Suggested solution doesn't allow debug tests and use JSON output simultaneously.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 7, 2017

I don't understand.

What do you mean by debuggable?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2017

Literally, I cannot debug a test with Delve.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2017

Sorry, can you spell out in more detail what you want to do?

Do you want to run Delve on a test, while looking at the JSON output from the test? Why is that an important use case?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2017

Yes, I want to debug my tests and have JSON output at the same time, not sure what I can add to this.

It's important because otherwise IDE or any other tool that provides debug support for tests won't be able to use brand new JSON output. Particularly I won't be able to fix a lot of issues that users asked us to fix in testing.

@ianlancetaylor ianlancetaylor changed the title Testing with -json option is not debuggable testing: testing with -json option is not debuggable Dec 7, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2017

Your IDE is going to be parsing the output of the program you are debugging? I'll just take your word for it. I can't picture that at all.

In that case, why not have your IDE pipe the test output through go tool test2json?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2017

Your IDE is going to be parsing the output of the program you are debugging?

Yes, it's not different in any way from the current situation when we have to parse messages like ===RUN.

In that case, why not have your IDE pipe the test output through go tool test2json?

Because of #22789, because it's much more complicated and because the issue not only about the IDE: it might be tools for remote debugging on continuous integrations servers or just for a user who wants to see json reports while debugging from the CLI.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2017

Is this still waiting for info?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 15, 2017

Well, yes, this is still waiting for info. Can you say precisely what change you want?

If the change you want is for tests to be able to generate JSON output without going through the test2json program, that is not going to happen for the reasons given over at #22996.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

Well, yes, this is still waiting for info. Can you say precisely what change you want?

I'm not asking for particular changes, I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

that is not going to happen for the reasons given over at #22996.

Having a test-binary that reports JSON would fix the issue automatically, but having such binary is not the goal, this is why it's different from #22996

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 15, 2017

I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

I'm sorry for asking the same question over and over again, but I honestly and genuinely do not understand what you mean. I can debug a test by running delve. Whether the test output JSON or not is entirely irrelevant to my debugging experience.

Let me try this a different way. Can you explain, in as much detail as possible, precisely what you do, what you expect to happen, and precisely what happens instead? Thanks.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

I can debug a test by running delve

This is what I want to know. How do you run it?

This is what I can imagine about running Delve and having JSON output:

  • dlv <binary> doesn't work because, as we know from #22996, a testing binary cannot produce JSON output.

  • dlv test -json <etc> doesn't work because dlv test is a shortcut for making testing binary and then running dlv <binary>.

  • dlv ./foo.test -test.v 2>&1 | go tool test2json is not a cross-platform solution (#22789). Also, it converts Delve's debug output, which is not desirable.

  • dlv go tool test2json ./foo.test -test.v produces JSON output but it doesn't stop at breakpoints, likely it tries to debug the wrong process

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

go tool test2json dlv <binary> should work but it converts Delve's output

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 15, 2017

When I'm running Delve, I don't really care whether the test is generating JSON output or not. I agree that it is hard to run Delve and simultaneously pipe the test program output, and only the test program output, into test2json.

Why do you want to do that?

It sounds like maybe you want something along the lines of gdb's MI support (https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html), which would permit this. I don't know whether Delve supports anything like that.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

When I'm running Delve, I don't really care whether the test is generating JSON output or not.

We care because we use Delve right now for debugging and we're trying to move to new JSON testing output in order to fix a bunch of different issues with tests output interpretation.

Delve has a protocol on top of tcp to chat with clients and we use this protocol. Still, Delve prints some debug information to stdout and it would be good not to mix it with tests output.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2017

No matter what, it is going to be necessary to separate Delve output from test output. So that sounds like something that needs to be changed in Delve, one way or another.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2017

Sorry, I don't understand. Could you please elaborate what kind of change in Delve you mean?

In general, I think it's not what third-party tools should worry about. Delve has nothing to do with tests or testing output. The only thing that it's responsible for is debugging some executable binary. The fact that its output will be piped to another tool is out of the scope of its interest.

The output format of a testing tool is the responsibility of the testing tool unless the format is configurable by clients like it's done in some java/ruby/js testing tools.

@cespare

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2017

Still, Delve prints some debug information to stdout and it would be good not to mix it with tests output.

Suppose the test binary could emit JSON. Wouldn't that be interleaved with with this "debug information"?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 16, 2017

@cespare it will, for sure, but I'll be able to distinguish JSON and non-JSON output. Or even complete JSON-test event and any other output.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 17, 2017

You said "Delve prints some debug information to stdout and it would be good not to mix it with tests output." The only possible way to reliably not mix that up is to modify Delve. You need to be able to distinguish Delve output from the output of the program that Delve is running.

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 17, 2017

is to modify Delve

Sorry, still unclear what the change you have in mind. What's the approach you suggest considering that only go tool test2json dlv <binary> may potentially work right now?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 17, 2017

And as I said your suggestion means that every tool that could work with test binaries and JSON output will have to do some changes. It is not scalable approach and it quite weird to require tools make hacks for tests.

I think we'll add profiling support in IDE in future and we'll do profiling tests. The suggested approach means that we'll face the very same problem again.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 17, 2017

I'm sorry, we are somehow failing to communicate. I still don't understand your problem, and you don't understand what I am saying.

As far as I can tell, you never answered my question above: "Can you explain, in as much detail as possible, precisely what you do, what you expect to happen, and precisely what happens instead?"

Let me also try a different question: what, precisely, would you like to change in Go?

@zolotov

This comment has been minimized.

Copy link
Contributor Author

commented Dec 21, 2017

Hm, I believe I answered the first question: I want to know a way to debug tests with JSON output without messing the output of debugger.

As for the second question about changes in Go, I think if I knew what precisely should be done, I'd made a PR :)

For now, I just declare the issue I faced and would like to have a solution for it.
#22996 might be a possible solution, so this is maybe considered as one more usecase for implementing it. If you have other ideas how to handle this – it's great.

By the way, I just tried to run something similar to go tool test2json dlv <binary>, it works almost fine but it finishes with 130 exit code. Do you know the reasons for such behavior of test2json tool?

That's the command I ran: /Users/zolotov/dev/go/bin/go tool test2json -t /path/to/dlv --listen=localhost:52757 --headless=true --api-version=2 --backend=default exec /path/to/compiled_test -test.v -test.run ^TestCounterClear$

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2018

Closing because I do not understand anything we can change in the Go distribution to fix this.

@golang golang locked and limited conversation to collaborators Mar 29, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.