StringIndexOutOfBoundsException in findClassesInPackage - Surefire/Maven - JDK 11 fails, JDK 8 succeeds.

[john-snell]

TestNG Version

7.1.0

Expected behavior

TestNG reads the package name correctly, succeeds in doing a substring

Actual behavior

TestNG keeps the package name from another class when doing a substring on current class, fails with error.

[ERROR] Caused by: java.lang.StringIndexOutOfBoundsException: begin 35, end 11, length 17
[ERROR]         at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
[ERROR]         at java.base/java.lang.String.substring(String.java:1874)
[ERROR]         at org.testng.internal.PackageUtils.findClassesInPackage(PackageUtils.java:112)
[ERROR]         at org.testng.xml.XmlPackage.initializeXmlClasses(XmlPackage.java:68)
[ERROR]         at org.testng.xml.XmlPackage.getXmlClasses(XmlPackage.java:59)

Having instrumented a copy of TestNG jar with print statements, I can see that this fails when a Jar has its files enumerated. Specifically jaxb-api-2.3.1.jar, which include a file 'module-info.class', which lacks a package. PackageUtils then tries to do a substring based on the wrong package, with a class name that has no package, and fails.

This does not occur during the JDK 8 build, as Java 8 doesn't cause the JAXB jars to be referenced in this part of the build, but Java 11 does.

My simple print statement:
org.testng.internal.PackageUtils#findClassesInPackage

for (URL url : dirs) {
  String protocol = url.getProtocol();
  if (!matchTestClasspath(url, packageDirName, recursive)) {
    continue;
  }
  System.err.println("url = " + url);

With Java 8:

url = file:/C:/projects/testng-failure-case/target/test-classes/
url = file:/C:/projects/testng-failure-case/target/classes/

With Java 11:

url = file:/C:/projects/testng-failure-case/target/test-classes/
url = file:/C:/projects/testng-failure-case/target/classes/
url = jar:file:/C:/Users/jsnell/.m2/repository/org/apache/logging/log4j/log4j-api/2.11.2/log4j-api-2.11.2.jar!/META-INF/versions/9/
url = jar:file:/C:/Users/jsnell/.m2/repository/org/apache/logging/log4j/log4j-core/2.11.2/log4j-core-2.11.2.jar!/META-INF/versions/9/
url = jar:file:/C:/Users/jsnell/.m2/repository/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar!/META-INF/versions/9/

Is the issue reproductible on runner?

Completely reproducible with Maven test case using attached source.
See output:
testng.failure.output.txt

Test case sample

testng-failure-case.zip

[krmahadevan]

@john-snell - I dont think TestNG is yet officially certified to work fine without any issues using JDK11.

[Randgalt]

When (if) will TestNG be Java 11 compatible? Java 11 has been LTS for almost 2 years

[krmahadevan]

@Randgalt - I don't have a definitive timeline that I can share with you. But TestNG needs to keep in account backward compatibility when we do such upgrades. The upgrade from JDK7 to JDK8 being the Default JDK itself took a lot of time (because a lot of our users still use older JDKs and upgrading TestNG to a newer JDK essentially means that we cause problems for them).

@cbeust @juherr please weigh in.

[juherr]

In fact, the target of the TestNG sources is supposed to be the previous Java LTS version. The current LTS version is Java 11 and the previous one is Java 8).
But TestNG is supposed to work at runtime for all supported versions after the target (Java 8, 11, 14).

I think there is a real issue if TestNG is failing at runtime with Java 11.

[Randgalt]

I think there is a real issue if TestNG is failing at runtime with Java 11.

It fails for us when we use a testng.xml file. We're completely blocked right now. We're going to try to remove the testng.xml if we can and hopefully that will work.

[pilophae]

The module-info.class present in for example the JAXB-API JAR triggers this with Java 11. Only if a recursive package name is set in the TestNG suite (<package name=".*">). When debugging, it looks like the packageName variable is kept from the previous file when reading the JAR file, causing out if index when assuming that module-info.class is also in a package.

[john-snell]

build-helper-maven-plugin appears to resolve the issue.
A co-worker came up with this variant of the test case, though it appears to be running the tests twice now.
testng-failure-case-working.zip