standarize/introduce Description links

[piotrturski]

currently the only way to interact with IDE is a Description. based on the Description IDE has to create nice looking tree and navigate to source when user clicks on this tree.

there are more and more custom Runners that introduces new things (parameters, nested tests etc). but still there is no standard, portable way to link to correct method. it would be good to separate what's displayed in the tree from what method that node is linked to. then intermediate nodes in parameter tests could be named as user/library creator wants yet still after double-click they will link to correct methods. same with leaf nodes

[kcooney]

Do you have any specific recommendations?

Edit: also would be good to have examples of open source runners that would benefit from this proposal (preferably, ones that are actively maintained)

[piotrturski]

nothing detailed. i would leave fDisplayName and stop adding there full class name and introduce new field link. that should be able to link to classes and method. maybe even it should link to specigic parameter (in this case we need source line and row number) so i'm not sure if it should be a string or something more complex. when IDEs start using it, the navigation will be much more procise and comfortable

[kcooney]

It sounds like the challenge here is to work with the IDEs to understand what they would need from the "link" to do what you want. Could you reach out to the Eclipse or IntelliJ developers and come up with a specific proposal?

[marcphilipp]

Interesting, I've also been thinking about this.

We could add some kind of marker interface, e.g. TestLocation, along with a few general purpose implementations:

• ClassLocation: a Java Class
• MethodLocation: a Java Method (and Class?)
• FileLocation: a File object along with an optional line number
• UrlLocation: a standard URL or URI object

Thoughts?

[jlink]

Where would the information, e.g. file and line number, come from? I know
that the groovy compiler adds all that stuff to the AST. Is there anything
like that available in Java via reflection or alike? My impression was that
IDEs add those information themselves since they usually will parse all
sources.

But maybe I misunderstood you and FileLocation would only be used for
file-based tests by external runners.

Johannes

2015-01-19 19:13 GMT+01:00 Marc Philipp notifications@github.com:

Interesting, I've also been thinking about this.

We could add some kind of marker interface, e.g. TestLocation, along with
a few general purpose implementations:

• ClassLocation: a Java Class
• MethodLocation: a Java Method (and Class?)
• FileLocation: a File object along with an optional line number)
• UrlLocation: a standard URL or URI object

Thoughts?

—
Reply to this email directly or view it on GitHub
#1070 (comment).

[piotrturski]

@kcooney i add posts on both intellij and eclipse devs forum. don't know if there will be any reposnose. probably we have to wait.

@jlink no, it's not IDE that adds it. when you thow and exception this informartion is in the stacktrace. it's added by the compiler. although during compilation you can explicitly request to remove those information almost noone does that

@marcphilipp currently IDEs do it differently by parsing Description. for example intellij doesn't navigate to methods with parameters. eclipse is a bit better here. for sure MethodLocation (with parameters) will be better than what is done currently. but in complex scenarios we have at least two lacations: test mathod and parameters for this specific invocation. mathod is in A.java and parameters are in data.txt row 50 column 16. sometimes both are in same file (@Parameterized, zohhak, junitParams, junit-dataprovider etc) but in different lines. and remember that there can be many different parameters set in the same line - so column is also important. And sometimes it's not important - if we generate parameters on the fly

[kcooney]

Note that the Description is created by the Runner, usually before the test is run, and is essentially immutable (in JUnit 5, we hope to make it immutable).

@piotrturski can you add links to the Eclipse and IntelliJ discussions here?

[jlink]

2015-01-19 22:10 GMT+01:00 Piotr Turski notifications@github.com:

@jlink https://github.com/jlink no, it's not IDE that adds it. when you
thow and exception this informartion is in the stacktrace. it's added by
the compiler. although during compilation you can explicitly request to
remove those information almost noone does that

So this only works after a test failure? How do you link test cases without
failures to their source?

Johannes

2015-01-19 22:10 GMT+01:00 Piotr Turski notifications@github.com:

@kcooney https://github.com/kcooney i add posts on both intellij and
eclipse devs forum. don't know if there will be any reposnose. probably we
have to wait.

@jlink https://github.com/jlink no, it's not IDE that adds it. when you
thow and exception this informartion is in the stacktrace. it's added by
the compiler. although during compilation you can explicitly request to
remove those information almost noone does that

@marcphilipp https://github.com/marcphilipp currently IDEs do it
differently by parsing Description. for example intellij doesn't navigate
to methods with parameters. eclipse is a bit better here. for sure
MethodLocation (with parameters) will be better than what is done
currently. but in complex scenarios we have at least two lacations: test
mathod and parameters for this specific invocation. mathod is in A.java and
parameters are in data.txt row 50 column 16. sometimes both are in same
file (@parameterized https://github.com/parameterized, zohhak,
junitParams, junit-dataprovider etc) but in different lines. and remember
that there can be many different parameters set in the same line - so
column is also important. And sometimes it's not important - if we generate
parameters on the fly

—
Reply to this email directly or view it on GitHub
#1070 (comment).

[piotrturski]

this information is always available:
http://stackoverflow.com/questions/7483421/how-to-get-source-file-name-line-number-from-a-java-lang-class-object

but after some reading it seems like the resolution is not goof enough :( there is probably no way to get column number and line number may also be a bit inaccurate. so i think all can be done is just better method linking, for example adding full signature instead of only name

[kcooney]

@piotrturski generally test methods don't have parameters, so the method name is sufficient. Is this specially for the case where the runner allows test methods to have parameters? If so, what information besides the signature would you need, and what will the caller do with this data?

In other words, can you provide a specific use case?

[jlink]

2015-01-20 16:35 GMT+01:00 Piotr Turski notifications@github.com:

this information is always available:

http://stackoverflow.com/questions/7483421/how-to-get-source-file-name-line-number-from-a-java-lang-class-object

but after some reading it seems like the resolution is not goof enough :(
there is probably no way to get column number and line number may also be a
bit inaccurate. so i think all can be done is just better method linking,
for example adding full signature instead of only name

One could create file and line info for assertions (failed and succeeded
ones). For test classes, methods and alike one could link to the most
specific java.lang.reflect-Object which would probably help the IDE to find
out about file and line number.

Johannes

[piotrturski]

@kcooney specific case is method with parameters. i'm aware that default runner uses methods without parameters. but if junit supports custom runners then better API for IDEs would be nice

[kcooney]

@piotrturski so for the case of methods with parameters, specifically what data would want that we don't provide in Description today?

Other than methods for parameters, what would you need from description "links"?

Note that if the intention is to use the data from the IDEs, then we would really need to work with the IDEs to design this. If we design in a vacuum the API will likely not be usable to the IDE teams.

Note I can imagine cases where the data passed a parameterized test could not be known until right before the test is run. Perhaps the data you want shouldn't be attached to the Description but instead added to the Failure or some other object that we pass to the listener.

[piotrturski]

in order to navigate to a method with parameters we need signatures. easiest way is to use JNI type signatures like (ILjava/lang/String;[I). details here: http://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/types.html
at the beginning i was thinking about simply keeping a reference to Method but IDEs may keep it in different classloaders or even different jvms.
for the same reason, re-running tests with its parameter may be possible only using serialization (i don't have other ideas at the moment) but requiring users to make their parameters serializable is unnecessary overkil and limitation