Support for using junit-vintage-engine as a substitute of JUnit 4

[Leo3418]

The components of JUnit 5 include the JUnit Platform and two test engines based on the platform: JUnit Jupiter and JUnit Vintage. The JUnit Platform is mainly for launching tests on a test engine. It has a TestEngine inteface that a test engine should implement. JupiterTestEngine and VintageTestEngine are the implementations of the interface from JUnit Jupiter and JUnit Vintage respectively.

JUnit Jupiter is what we would generally mean by "JUnit 5". The assert* methods in the Assertions class, the @Test annotation, and many other commonly-used JUnit 5 classes and methods all come from the org.junit.jupiter.api package provided by Jupiter.

JUnit Vintage is a test engine that allows JUnit 4 and 3 tests to be launched by the JUnit Platform. It is released with all other JUnit 5 modules together, but it is not frequently used by projects that use JUnit 5 (they often only use JUnit Jupiter).

Because JUnit 4 itself already provides the functionality to launch test cases, this means we have two different ways to launch JUnit 4 tests:

• Use dev-java/junit:4 to launch them (which is what JAVA_TESTING_FRAMEWORKS="junit-4" would do)
• Use JUnit Platform and JUnit Vintage to launch them (which is the option being evaluated here)

Pros of Supporting JUnit Vintage

One perk of using the JUnit Platform to launch JUnit 4 tests is finer granularity in test selection. Tests can be selected using directories, packages, classes and methods. Test classes can be included or excluded using regular expressions and package names. More details are available in the upstream's documentation.

Compared to the JUnit 4 test launcher, the JUnit Platform is better in providing detailed messages and reports with regards to test execution and results. Recently, we have a new pull request to ::gentoo (gentoo/gentoo#23930) adding dev-java/javax-mail. This package's test suite would hang indefinitely on Java 11. JUnit 4 will get stuck with the following output without indicating the test it has trouble finishing, making it very difficult for debugging:

>>> Test phase: dev-java/javax-mail-1.6.2
/var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail/src/test/java /var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail
/var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail
 * Compiling ...
warning: [options] bootstrap class path not set in conjunction with -source 8
Note: src/test/java/javax/mail/internet/MailDateFormatTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning
JUnit version 4.13.2-SNAPSHOT
.........................................................................................................................................................ContentTypeCleaner
.E.E.E..........

But with JUnit Vintage, the tests hang with the following output (as long as --details=verbose is set in the command-line arguments to ConsoleLauncher), explicitly telling us that the problematic test is className = 'com.sun.mail.smtp.SMTPUtf8Test', methodName = 'testUtf8From':

JUnit Vintage output

>>> Test phase: dev-java/javax-mail-1.6.2
/var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail/src/test/java /var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail
/var/tmp/portage/dev-java/javax-mail-1.6.2/work/javamail-JAVAMAIL-1_6_2/mail
 * Compiling ...
warning: [options] bootstrap class path not set in conjunction with -source 8
Note: src/test/java/javax/mail/internet/MailDateFormatTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning
 * No suitable function found for framework junit-vintage
 * Running JUnit 5 tests ...

Thanks for using JUnit! Support its development at https://junit.org/sponsoring

Test plan execution started. Number of static tests: 772
╷
├─ JUnit Vintage
│  ├─ SMTPWriteTimeoutTest
│  │  ├─ test
│  │  │       tags: []
│  │  │   uniqueId: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPWriteTimeoutTest]/[test:test(com.sun.mail.smtp.SMTPWriteTimeoutTest)]
│  │  │     parent: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPWriteTimeoutTest]
│  │  │     source: MethodSource [className = 'com.sun.mail.smtp.SMTPWriteTimeoutTest', methodName = 'test', methodParameterTypes = '']
│  │  │   duration: 1659 ms
│  │  │     status: ✔ SUCCESSFUL
│  └─ SMTPWriteTimeoutTest finished after 1660 ms.
│  ├─ SMTPUtf8Test
│  │  ├─ testUtf8UserNamePlain
│  │  │       tags: []
│  │  │   uniqueId: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]/[test:testUtf8UserNamePlain(com.sun.mail.smtp.SMTPUtf8Test)]
│  │  │     parent: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]
│  │  │     source: MethodSource [className = 'com.sun.mail.smtp.SMTPUtf8Test', methodName = 'testUtf8UserNamePlain', methodParameterTypes = '']
│  │  │   duration: 8 ms
│  │  │     status: ✔ SUCCESSFUL
│  │  ├─ testUtf8UserNameNoAllowUtf8
│  │  │       tags: []
│  │  │   uniqueId: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]/[test:testUtf8UserNameNoAllowUtf8(com.sun.mail.smtp.SMTPUtf8Test)]
│  │  │     parent: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]
│  │  │     source: MethodSource [className = 'com.sun.mail.smtp.SMTPUtf8Test', methodName = 'testUtf8UserNameNoAllowUtf8', methodParameterTypes = '']
│  │  │   duration: 3 ms
│  │  │     status: ✔ SUCCESSFUL
│  │  ├─ testUtf8UserName
│  │  │       tags: []
│  │  │   uniqueId: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]/[test:testUtf8UserName(com.sun.mail.smtp.SMTPUtf8Test)]
│  │  │     parent: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]
│  │  │     source: MethodSource [className = 'com.sun.mail.smtp.SMTPUtf8Test', methodName = 'testUtf8UserName', methodParameterTypes = '']
│  │  │   duration: 2 ms
│  │  │     status: ✔ SUCCESSFUL
│  │  ├─ testUtf8From
│  │  │       tags: []
│  │  │   uniqueId: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]/[test:testUtf8From(com.sun.mail.smtp.SMTPUtf8Test)]
│  │  │     parent: [engine:junit-vintage]/[runner:com.sun.mail.smtp.SMTPUtf8Test]
│  │  │     source: MethodSource [className = 'com.sun.mail.smtp.SMTPUtf8Test', methodName = 'testUtf8From', methodParameterTypes = '']

Cons of Supporting JUnit Vintage

• JUnit Vintage still depends on dev-java/junit:4, and it depends on the JUnit Platform too. This means, comparing to using JUnit 4 directly, a few extraneous dependencies must be pulled if JUnit Vintage is used.
• Providing two ways to accomplish the same task can cause confusions to ebuild authors as to which test launch method to use.