[NETBEANS-6] Adding Support for JUnit 5

[mcdonnell-john]

Main Features include:

• Added JUnit 5 Library - Using latest 5.3.1 release.
• Added JUnit 5 Templates
• New Test classes default to JUnit 5 if JDK >= 8 & JUnit 4 not already used.
• Test Classes when JUnit 5 is found on the class path(Maven) use JUnit 5.
• Run/Debug single test method support included. Checks when Using JUnit 5 of the method has an annotation that inherits from @testable
• Updated JUnit IDE hints for JUnit 5 - i.e. Assertions as opposed to Assert and ordering of arguements.
• Informational message to users when creating a Test Suite with Junit 5 - Currently not supported with Jupiter, message refers users to switch to the Vintage Engine.

[sormuras]

Nice to see JUnit 5 support in NB, soon! ;-)

Using Testable as the signal to show users "testable" elements is good.

Just scrolled through the PR and didn't find a single "junit-platform-launcher" snippet. How does NetBeans execute tests? Via the command line? Or Maven/Surefire?

[mcdonnell-john]

Hi @sormuras,

So I toy-ed around for a while looking to see if I could implement an EngineDiscoveryRequest but couldn't get it to work, and then went back to the @testable check based on its JavaDocs and left it at that for now.

When using Maven based projects, Netbeans uses Maven/Surefire - I added a quick check for Surefire >2.22.0 for supporting individual test methods.
I don't use Ant for any project, so would need help from someone who does to see what's needed to support changes for Ant.

[sormuras]

I see. So, at the moment, users see something like "Run test..." in the IDE/editor and when click, a Maven run tailored to the action is executed. Correct? If yes, it's good starting point.

You could use the junit-platform-console launcher, like I do here: https://github.com/sormuras/junit-platform-maven-plugin/tree/master/src/main/java/de/sormuras/junit/platform/maven/plugin -- then you are less dependent on how Surefire finds and executes tests.

Regarding Ant and its configuration in NB ... methinks, it would need to include junit-platform-console and use the provided launcher.

Or...

You use the Launcher API like described here: https://junit.org/junit5/docs/current/user-guide/#launcher-api

How can I test-drive this PR? Checkout your "apache/incubator-netbeans" clone, switch to branch feature/JUnit5 and "Press Play on Tape"?

[sormuras]

Answering my last question on my own: http://netbeans.apache.org/download/index.html#source

[mcdonnell-john]

Thanks for the comments.

NetBeans is closely tied to maven, and for JUnit surefire - I tried to keep as close to the existing solution that exists for JUnit 4 as I could to keep the changes as simple as possible.

I think the biggest issue with this is the fact that surefire doesn't support the @DisplayName annotation, since the reporting in NetBeans also uses the output from Surefire from what I have seen. - So that's a gap we have with the maven support for JUnit 5 in NetBeans

Yes,
"How can I test-drive this PR? Checkout your "apache/incubator-netbeans" clone, switch to branch feature/JUnit5 and"... - run "ant tryme"

[junichi11]

Please add the template file to the licenseinfo.xml if don't add the license header to template files. e.g. see https://github.com/apache/incubator-netbeans/blob/7b3b31b62e5fa95f7fbf6a60cbcc5b0838f42740/ide/xml/licenseinfo.xml#L23-L31

[mcdonnell-john]

Hi @junichi11 Thanks for spotting this.

Just a note on this point though, the other licenses aren't covered in a licenseinfo.xml file, they are infact covered via the rat exclusions file: https://github.com/apache/incubator-netbeans/blob/7b3b31b62e5fa95f7fbf6a60cbcc5b0838f42740/nbbuild/rat-exclusions.txt#L188. So my new template file should be covered by this, as opposed to a licenseinfo.xml entry.

I added a new logo to differentiate JUnit 5 and the old JUnit version so will definitely add a reference to that as that was missed and commit that shortly.

[junichi11]

I see. I suppose that we should move them to the licenseinfo.xml if possible because we have to verify rat-exclusions and licenseinfo.xml :)

[junichi11]

Please add the template file to the licenseinfo.xml.

[mcdonnell-john]

same comment as above.

[sormuras]

Regarding @DisplayName and Surefire: working on it already https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1546

[sormuras]

FYI: there is an issue with Surefire keeps using version 5.2 of JUnit (JUnit Platform 1.2.0) although you override the version in a pom.xml file. See https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1564 for details.

[mcdonnell-john]

Regarding @DisplayName and Surefire: working on it already https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1546

Cool, When this comes in, then we may have to look at the Test Results window in NetBeans to show the names explicitly.

[sormuras]

If stars align, it should work out-of-the-box. Just update to the yet to be released Surefire version and the ReportEntry object will report the display names instead of the "legacy reporting names", i.e. the name of the test methods and classes.

[geertjanw]

Great, if I can do something to try this out, would be great -- maybe you could update this page with the list of features provided for JUnit 5 here: https://cwiki.apache.org/confluence/display/NETBEANS/Miscellaneous

[geertjanw]

Any reason not to merge this, what still needs to be done, etc?

[mcdonnell-john]

Any reason not to merge this, what still needs to be done, etc?

Hi, I don't think there is.

[geertjanw]

Can you add screenshots to illustrate these features to that Miscellaneous page?

[mcdonnell-john]

Will do - should be able to get to that on Sunday...

[Saljack]

Hi, why JUnit BOM is not used in pom.xml for versions and dependencies? It looks better and more clear in my oppinion?

  </dependencies>
    <!--JUNIT 5-->
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-params</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>5.3.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

[mcdonnell-john]

Hi, why JUnit BOM is not used in pom.xml for versions and dependencies? It looks better and more clear in my oppinion?

Because the current NetBeans implementation of adding libraries to an existing pom doesn't support that, as far as I could see. I tried to limit my changes to re-use a lot of what already worked with JUnit 4.

It's also a matter of taste, and to me its not a big deal.

[geertjanw]

Great, and thanks for the screenshots (https://cwiki.apache.org/confluence/display/NETBEANS/Miscellaneous). What is left to be done, for this release, and for subsequent releases?

[mcdonnell-john]

HI @geertjanw I cant think of anything else, I mostly focused on Maven support and maybe some ant stuff was handle there, but there probably some work there if someone who uses ant wants to take on.

I think for this release we should be okay, and we'll need to update the Test Specs for JUnit testing for NetCat to help test this a little more.

Moving forward, for subsequent releases:
What ever is missing for Ant support,
Better test suite generation - when Jupiter supports it

Apart from that I don't see anything as Surefire should be fixing their display name annotation handling issue, if that doesn't need any additional changes then great, if not we can include that in a future release.

[geertjanw]

Great. Let's merge if no one objects 24 hours from now.

[Tibor17]

We want to cut a new Surefire Version 2.22.1 for JUnit5 5.2.0+.
This will support 5.3.0 too of course and Java 11.
We have decided that version 3.0.0-M1 will adopt Display Name.
We won't to postpone the release 2.22.1 too much.

[geertjanw]

I took a look at JUnit 5 and Ant in NetBeans, some thoughts here:

https://blogs.apache.org/netbeans/entry/junit-5-apache-ant-and

[sormuras]

Probably the best solution for Ant users in Apache NetBeans would be to be able to continue to use JUnit 4, rather than JUnit 5, since the latter is a total nightmare to set up in the context of Ant, as described above. No need to endure nightmares...

Have you tried the all-in-one-fat-jar solution with Ant? Download junit-platform-console-standalone and that's all.

Copied from the User Guide

junit-platform-console-standalone
An executable JAR with all dependencies included is provided at Maven Central under the
junit-platform-console-standalone directory. See Console Launcher for details.

[geertjanw]

Ah, OK, that's great.

Could we bundle that with Apache NetBeans, in the same way that we're currently bundling jupiter-api, jupiter-params, and jupiter-engine? And could we remove those three and replace it with the fat JAR? Is it licensed in the same way?

[sormuras]

Could we bundle that with Apache NetBeans,...

Perhaps. It's a "all-in" solution. Could Maven projects still refer to the modules and use them while Ant projects get the junit-platform-console-standalone support?

Is it licensed in the same way?

It states:

Open Source Licenses
====================

This product may include a number of subcomponents with separate
copyright notices and license terms. Your use of the source code for
these subcomponents is subject to the terms and conditions of the
subcomponent's license, as noted in the LICENSE-<subcomponent>.md
files.

It contains the following licences:

• Eclipse Public License - v 1.0 (junit 3 + 4)
• Eclipse Public License - v 2.0 (junit 5)
• BSD License (hamcrest)
• Apache License 2.0 (picocli, univocity-parsers)

[sormuras]

@geertjanw Here is an example of a "patched" Ant installation that allows Ant projects to be configured hassle-free:

https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-ant

See https://github.com/junit-team/junit5-samples/blob/master/junit5-jupiter-starter-ant/build.sh#L6-L20 for how to "install" the junit-platform-console-standalone.jar into Ant (as a lib) and here is how you may use the built-in task and the console launcher side-by-side: https://github.com/junit-team/junit5-samples/blob/master/junit5-jupiter-starter-ant/build.xml#L35-L61

[geertjanw]

Yes, I've been studying this one a lot, it's great and very useful:

https://github.com/junit-team/junit5-samples/blob/master/junit5-jupiter-starter-ant/build.xml

And the org.junit.platform.console.ConsoleLauncher works fine, produces results, etc. The only question is how to hook it into the Results Window in NetBeans, which is done very well for Surefire, which is why JUnit 5 with Maven in NetBeans works so smoothly in this regard.

[sormuras]

[...] which is done very well for Surefire [...]

By a listener that Surefire calls when tests are executed?

Nevertheless and as you have written in your blog, NetBeans should use the JUnit Platform Launcher API to start test runs. That's the sole purpose of it -- no need to depend on a plugin of a specific build tool that tries to "find and execute" tests in a "custom" way. Copied from the beginning of the User Guide:

The JUnit Platform serves as a foundation for launching testing frameworks on the JVM. It also defines the TestEngine API for developing a testing framework that runs on the platform. Furthermore, the platform provides a Console Launcher to launch the platform from the command line and build plugins for Gradle and Maven as well as a JUnit 4 based Runner for running any TestEngine on the platform.

https://junit.org/junit5/docs/current/user-guide/index.html#overview-what-is-junit-5

And every new TestEngine, wrapping an existing test framework (mainrunner, TestNG, Spock, JQWIK, ... https://github.com/junit-team/junit5/wiki/Third-party-Extensions#junit-platform-test-engines) or one that is invented tomorrow (JUnit 6 "aka Saturn") will be integrated in your build tool right-away. Zero effort.