Facilitate discovery of test cases for platform clients using sources only

[mmerdes]

Motivation

Some clients of the JUnit 5 platform, notably IDEs such as IntellilJ IDEA operate only on sources
for their test detection. Thus, they cannot use the full discovery mechanism of the JUnit platform as the latter relies on compiled classes.

Therefore a mechanism is needed to detect test cases by analyzing the source code only.

In this context, a test case is anything a particular test engine would like to execute as a test. For the JUnit Jupiter Engine this includes test and test factory methods. Knowing about test cases early before compilation is important to interactive platform clients such as IDEs as it can vastly improve the user experience by providing test running means within the source code and in context menus.

Precondition

It is assumed that said interactive platform clients possess a means to find meta-annotations in non-compiled source code. This must obviously include the ability to detect meta-meta-annotations etc.

Proposal

Introduce an annotation such as @Testable that all engines can use to denote which types or methods are "testable". As this notion of being testable solely depends on the interpretation of a (custom) engine the IDE cannot deduce this testability in general for all possible engines.

Such a @Testable annotation will then be used to meta-annotate the engine's 'normal' test annotation(s). Thus this approach does not put an additional burden on test authors as they can still use the familiar test annotations. An IDE however can analyze all meta-annotations of the standard test annotations to find the test cases without being required to have any intimate knowledge of a particular test engine's annotations.

For the JUnit Jupiter engine such normal test annotations making use of @Testable would be @Test, @TestFactory, and the planned @Step.

According to this proposal the @Test annotation, for example, would be meta-annotated in the following way:

@Target({ ElementType.ANNOTATION_TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@API(Stable)
@Testable
public @interface Test {
}

Note that this approach would also be feasible for inheritance based test engines. These engines could put the @Testable annotation on their respective base classes.

Requirements

• If a class (including static nested classes or inner classes) is not meta-annotated with @Testable but contains a method that is meta-annotated with @Testable, the class must be considered to be a "testable" class.
• If annotation hierarchies (containing @Testable) are present in compiled byte code in JARs in the user's classpath, IDEs must also take such annotation hierarchies into consideration when used in conjunction with annotation processing for not-yet-compiled source code.

Limitations

This approach would not work for test engines unwilling or unable to use annotations at all. Examples include non-JVM languages or file formats such as XML or engines relying on naming conventions only, such as the JUnit 3 naming convention.

Restrictions for TestEngine Implementations

A TestEngine must not in any way perform discovery based on the presence of @Testable. In terms of discovery, the presence of @Testable should only be meaningful to clients such as IDEs. TestEngine implementations will therefore be required to discover tests based on information specific to that test engine (e.g., annotations specific to that test engine).

Open Questions

• If a class is annotated or meta-annotated with @Testable but contains methods that are not annotated or meta-annotated with @Testable but should be considered test methods (e.g. some TestNG-like use cases), will IntelliJ consider the methods to be "testable"?
  • ✅ @akozlova says "yes". See comments for details.

[sbrannen]

LGTM. Thanks for writing this up! 👍

FYI: I introduced a Restrictions for TestEngine Implementations section.

[bechte]

With this approach, we'd introduce some kind of "shadow discovery", that is limited to static code analysis. While this seems to be cool for IDEs to support the test writers, I'm unhappy to limit ourselves to static test discovery. There are plans for dynamic tests and dynamic test discovery. In order to support those, the IDE will have to execute the discovery anyways. Moreover, programming models that do not rely on Java code will also have to be executed and ask in order to discover their tests.

Having said that, I think we could introduce the proposed @Testable meta-annotation, but only as a good will , not as a reliable detection mechanism.

One question that comes along with it: In order to be consistent, we would have to annotate test classes as well (like we do with @Nested for nested test classes). This is something that I would like to avoid for top level classes.

[marcphilipp]

@bechte We are planning to annotate @TestFactory with @Testable as well. So there shouldn't be any problems with dynamic tests in Jupiter.

We probably should add that if any method in a class or any inner class is meta-annotated with @Testable it must be considered as being annotated with @Testable as well, right?

[mmerdes]

@akozlova
What do you think of the current state of the proposal?
If you have any ideas for improvements please let us know.

[sbrannen]

We probably should add that if any method in a class or any inner class is meta-annotated with @Testable it must be considered as being annotated with @Testable as well, right?

Yes. That is exactly what I meant when I wrote the following:

If a class is not meta-annotated with @Testable but contains a method that is meta-annotated with @Testable, will IntelliJ consider the class to be a "testable" class?

It is phrased as a question and targeted at IntelliJ... but only because it was meant for Anna specifically. 😉

In general, I intended that to be a requirement.

[sbrannen]

Update: in order to avoid further confusion, I moved two of the open questions to a new Requirements section.

[akozlova]

Guys,

I think the proposal would work, and of course all meta-annotations would be processed (Testable as well as Test, etc). IDE could check if a class contains testable method, then the class would be automatically be testable - that's how it works now - no problem here.

I'll try to check if I can write an annotation processor so this would help for an initial run when there is no compiled code - but I can't promise if this would work :/

What can I tell from the spek (Kotlin test framework): first people are trilled to have no methods and write the tests like sentences and then they are disappointed that they can't run parts of that test suites as there is nothing runnable there. Hopefully, @testable would make TestEngine writers to think about that kind of problems.

[sbrannen]

Hi @akozlova,

That all sounds good.

Thanks for the feedback!

[sbrannen]

@akozlova, just one more thing...

Can you please comment on the following open question?

• If a class is annotated or meta-annotated with @Testable but contains methods that are not annotated or meta-annotated with @Testable but should be considered test methods (e.g. some TestNG-like use cases), will IntelliJ consider the methods to be "testable"?

[akozlova]

I think yes, for non-private methods. In the worst case the engine won't be able to run it.

[sbrannen]

OK. Thanks!

[marcphilipp]

I'll try to check if I can write an annotation processor so this would help for an initial run when there is no compiled code - but I can't promise if this would work :/

@akozlova If I understood you correctly, you need to check if you can write such an annotation processor based on the AST first, before you can tell whether this proposal is feasible, right? If so, how long do you think you will need for that?

[akozlova]

I thought about asm instrumentation so I would have class bytes and would define them in my classloader to check if junit can find tests there, and what tests would be found. I have no idea of timing, sorry. Looks pretty simple but the devil is in the details ...

This won't work for initial case when user checks out a new project and want to start a test right after that. So I think that the annotation would be feasible anyway.