Make nested groups work for JDK >= 18

[duponter]

Testing Problem

We are currently looking to upgrade our application from Java 17 to Java 21.
While our application starts up without issues on Java 21, our unit tests do not complete successfully.
More specifically, most (not all) of our jqwik unit tests fail.

Reproduction

I could, however, reproduce the issue with a simple test case:

class JqwikJdk21Test {
    @Group
    class InGroup {

        @Property
        void runsWithAnnotations(@ForAll @NumericChars @StringLength(9) String input) {
            assertThat(input).hasSize(9).containsOnlyDigits();
        }


        @Property
        void runsWithProvideFromOuterClass(@ForAll("numericStrings") String input) {
            assertThat(input).hasSize(9).containsOnlyDigits();
        }
    }

    @Property
    void runsWithProvideFromSameClass(@ForAll("numericStrings") String input) {
        assertThat(input).hasSize(9).containsOnlyDigits();
    }

    @Provide
    static Arbitrary<String> numericStrings() {
        return Arbitraries.strings().numeric().ofLength(9);
    }
}

I am using following versions:

• openjdk 21.0.1 2023-10-17 LTS
• org.junit.jupiter:junit-jupiter-api:5.10.0
• org.junit.jupiter:junit-jupiter-engine:5.10.0
• net.jqwik:jqwik:1.8.1
• net.jqwik:jqwik-engine:1.8.1
• org.assertj:assertj-core:3.24.2

When running this test, runsWithProvideFromOuterClass fails. However, the same test runsWithProvideFromSameClass succeeds.
Additionally, when reverting to Java 17, all tests succeed.

Following stacktrace is printed for runsWithProvideFromOuterClass ran with Java 21.

java.lang.IllegalArgumentException: Method [static net.jqwik.api.Arbitrary org.jqwik.jdk21.JqwikJdk21Test.numericStrings()] cannot be invoked on target [org.jqwik.jdk21.JqwikJdk21Test$InGroup@57a3e26a].

	at net.jqwik.engine.support.JqwikReflectionSupport.invokeMethodPotentiallyOuter(JqwikReflectionSupport.java:236)
	at net.jqwik.engine.properties.ProviderMethod.invokeProviderMethod(ProviderMethod.java:58)
	at net.jqwik.engine.properties.ProviderMethod.lambda$arbitrarySuppliers$1(ProviderMethod.java:67)
	at net.jqwik.engine.properties.ProviderMethod.lambda$invoke$0(ProviderMethod.java:54)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at net.jqwik.engine.properties.ProviderMethod.mapSet(ProviderMethod.java:112)
	at net.jqwik.engine.properties.ProviderMethod.invoke(ProviderMethod.java:54)
	at net.jqwik.engine.properties.InstanceBasedSubtypeProvider.lambda$resolveFromGeneratorName$7(InstanceBasedSubtypeProvider.java:108)
	at java.base/java.util.Optional.map(Optional.java:260)
	at net.jqwik.engine.properties.InstanceBasedSubtypeProvider.resolveFromGeneratorName(InstanceBasedSubtypeProvider.java:108)
	at net.jqwik.engine.properties.InstanceBasedSubtypeProvider.apply(InstanceBasedSubtypeProvider.java:66)
	at net.jqwik.engine.properties.PropertyMethodArbitraryResolver.apply(PropertyMethodArbitraryResolver.java:11)
	at net.jqwik.engine.properties.PropertyMethodArbitraryResolver.forParameter(PropertyMethodArbitraryResolver.java:37)
	at net.jqwik.engine.execution.CachingArbitraryResolver.lambda$forParameter$0(CachingArbitraryResolver.java:25)
	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1228)
	at net.jqwik.engine.execution.CachingArbitraryResolver.forParameter(CachingArbitraryResolver.java:23)
	at net.jqwik.engine.properties.ExhaustiveShrinkablesGenerator.resolveParameter(ExhaustiveShrinkablesGenerator.java:31)
	at net.jqwik.engine.properties.ExhaustiveShrinkablesGenerator.lambda$forParameters$0(ExhaustiveShrinkablesGenerator.java:19)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at net.jqwik.engine.properties.ExhaustiveShrinkablesGenerator.forParameters(ExhaustiveShrinkablesGenerator.java:20)
	at net.jqwik.engine.execution.CheckedProperty.createOptionalExhaustiveShrinkablesGenerator(CheckedProperty.java:218)
	at net.jqwik.engine.execution.CheckedProperty.getOptionalExhaustive(CheckedProperty.java:251)
	at net.jqwik.engine.execution.CheckedProperty.chooseGenerationMode(CheckedProperty.java:204)
	at net.jqwik.engine.execution.CheckedProperty.createGenericProperty(CheckedProperty.java:141)
	at net.jqwik.engine.execution.CheckedProperty.check(CheckedProperty.java:67)
	at net.jqwik.engine.execution.PropertyMethodExecutor.executeProperty(PropertyMethodExecutor.java:90)
	at net.jqwik.engine.execution.PropertyMethodExecutor.executeMethod(PropertyMethodExecutor.java:69)
	at net.jqwik.engine.execution.PropertyMethodExecutor.lambda$execute$0(PropertyMethodExecutor.java:49)
	at net.jqwik.api.lifecycle.AroundPropertyHook.lambda$static$0(AroundPropertyHook.java:46)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
	at net.jqwik.api.lifecycle.PropertyExecutor.executeAndFinally(PropertyExecutor.java:39)
	at net.jqwik.engine.hooks.lifecycle.PropertyLifecycleMethodsHook.aroundProperty(PropertyLifecycleMethodsHook.java:56)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
	at net.jqwik.engine.hooks.statistics.StatisticsHook.aroundProperty(StatisticsHook.java:37)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$0(HookSupport.java:26)
	at net.jqwik.engine.hooks.lifecycle.AutoCloseableHook.aroundProperty(AutoCloseableHook.java:13)
	at net.jqwik.engine.execution.lifecycle.HookSupport.lambda$wrap$1(HookSupport.java:31)
	at net.jqwik.engine.execution.PropertyMethodExecutor.execute(PropertyMethodExecutor.java:47)
	at net.jqwik.engine.execution.PropertyTaskCreator.executeTestMethod(PropertyTaskCreator.java:166)
	at net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$1(PropertyTaskCreator.java:51)
	at net.jqwik.engine.execution.lifecycle.CurrentDomainContext.runWithContext(CurrentDomainContext.java:28)
	at net.jqwik.engine.execution.PropertyTaskCreator.lambda$createTask$2(PropertyTaskCreator.java:50)
	at net.jqwik.engine.execution.pipeline.ExecutionTask$1.lambda$execute$0(ExecutionTask.java:31)
	at net.jqwik.engine.execution.lifecycle.CurrentTestDescriptor.runWithDescriptor(CurrentTestDescriptor.java:17)
	at net.jqwik.engine.execution.pipeline.ExecutionTask$1.execute(ExecutionTask.java:31)
	at net.jqwik.engine.execution.pipeline.ExecutionPipeline.runToTermination(ExecutionPipeline.java:82)
	at net.jqwik.engine.execution.JqwikExecutor.execute(JqwikExecutor.java:46)
	at net.jqwik.engine.JqwikTestEngine.executeTests(JqwikTestEngine.java:70)
	at net.jqwik.engine.JqwikTestEngine.execute(JqwikTestEngine.java:53)

Possible solution

When a property test requires a Provide method from outside its own class, at some point JqwikReflectionSupport#getOuterInstance is called. This method calls the deprecated private method makeAccessible, but swallows any exceptions, returning an empty Optional instead.
Probably the stacktrace of the swallowed exception will give a clue to fix this issue.

[jlink]

@duponter Thanks for creating this issue.
Obviously Java’s revenge for me not having bothered to build with 21 in CI ;-/

I’ll track the problem down when I’m back from holidays. So not earlier than next week.

[duponter]

@jlink I'm sure that won't happen a second time ;-) Enjoy your holidays.

[jlink]

I cannot replicate the problem using tem-21.0.1 form IntelliJ or Gradle.

Can you try with "1.8.2-SNAPSHOT" release? Maybe some of the underlying lib updates I did solved the problem.

If you still see failing test, please specify how you start the tests.

[duponter]

I updated my libraries to the latest versions as well and built 1.8.2-SNAPSHOT from your main branch, however the problem persists.

Even on the command line (without Intellij or Maven interference) this single test fails (the other ones succeed). This is my command (replace <USER_HOME> with the parent of your .m2 directory):

java -jar junit-platform-console-standalone-1.10.1.jar --class-path target/test-classes
:<USER_HOME>/.m2/repository/net/jqwik/jqwik-api/1.8.2-SNAPSHOT/jqwik-api-1.8.2-SNAPSHOT.jar
:<USER_HOME>/.m2/repository/net/jqwik/jqwik-engine/1.8.2-SNAPSHOT/jqwik-engine-1.8.2-SNAPSHOT.jar
:<USER_HOME>/.m2/repository/org/assertj/assertj-core/3.24.2/assertj-core-3.24.2.jar
 --select-class org.jqwik.jdk21.JqwikJdk21Test

This is the output:

╷
├─ JUnit Jupiter ✔
├─ JUnit Vintage ✔
├─ JUnit Platform Suite ✔
└─ jqwik (JUnit Platform) ✔
   └─ JqwikJdk21Test ✔
      ├─ InGroup ✔
      │  ├─ runsWithProvideFromOuterClass ✘ Method [static net.jqwik.api.Arbitrary org.jqwik.jdk21.JqwikJdk21Test.numericStrings()] cannot be invoked on target [org.jqwik.jdk21.JqwikJdk21Test$InGroup@7ce97ee5].
      │  └─ runsWithAnnotations ✔
      └─ runsWithProvideFromSameClass ✔

Failures (1):
  jqwik (JUnit Platform):JqwikJdk21Test:InGroup:runsWithProvideFromOuterClass
    MethodSource [className = 'org.jqwik.jdk21.JqwikJdk21Test$InGroup', methodName = 'runsWithProvideFromOuterClass', methodParameterTypes = 'java.lang.String']
    => java.lang.IllegalArgumentException: Method [static net.jqwik.api.Arbitrary org.jqwik.jdk21.JqwikJdk21Test.numericStrings()] cannot be invoked on target [org.jqwik.jdk21.JqwikJdk21Test$InGroup@7ce97ee5].
       net.jqwik.engine.support.JqwikReflectionSupport.invokeMethodPotentiallyOuter(JqwikReflectionSupport.java:236)
       net.jqwik.engine.properties.ProviderMethod.invokeProviderMethod(ProviderMethod.java:58)
       net.jqwik.engine.properties.ProviderMethod.lambda$arbitrarySuppliers$1(ProviderMethod.java:67)
       net.jqwik.engine.properties.ProviderMethod.lambda$invoke$0(ProviderMethod.java:54)
       java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
       java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
       java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
       java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
       java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
       java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
       [...]

Test run finished after 168 ms
[         6 containers found      ]
[         0 containers skipped    ]
[         6 containers started    ]
[         0 containers aborted    ]
[         6 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         0 tests skipped         ]
[         3 tests started         ]
[         0 tests aborted         ]
[         2 tests successful      ]
[         1 tests failed          ]

I get a similar output when running with jqwik 1.8.1.

For reference, this is the JDK I use.

❯ java --version
openjdk 21.0.1 2023-10-17 LTS
OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode)

[jlink]

Which platform (Linux/win/macOS) are you running?

[duponter]

I am running on macOS Sonoma 14.1.
I also got the error when executing my tests using the official Maven Docker image with tag 3.9.5-eclipse-temurin-21

[jlink]

I still fail to replicate. I assume there's something else going on here, maybe related to run parameters or such.

I added Java 21 to the build matrix and everything seems to be fine: https://github.com/jqwik-team/jqwik/actions/runs/6795859226

Can you set up a dedicated repo to isolate your config that I can clone and run?

[jlink]

I just republished "1.8.2-SNAPSHOT" with some logging in case of a failed access to the outer object. Maybe you get a more expressive error message now.

[duponter]

I created a repository with a Github action that builds and produces the error:
https://github.com/duponter/jqwik-jdk21-test/actions/workflows/maven.yml

[duponter]

Also, my assumption was wrong. There is no exception that is being swallowed. So no additional logging is printed.

Instead, makeAccessible isn't even called at all, because inner.getClass().getDeclaredFields() returns an empty field array.

private static Optional<Object> getOuterInstance(Object inner) {
   return Arrays
	.stream(inner.getClass().getDeclaredFields()) // returns empty array, which yields an empty stream
	.filter(field -> field.getName().startsWith("this$"))
	.findFirst()
	.map(field -> {
		try {
			return makeAccessible(field).get(inner);
		} catch (SecurityException | IllegalArgumentException | IllegalAccessException ex) {
			String message = String.format("Could not access outer instance of %s." +
											"%nReason: %s", inner, ex);
			LOG.warning(message);
			return Optional.empty();
		}
	});
}

[jlink]

Now I can replicate it. Thanks for the effort!

[jlink]

So the problem is that .getDeclaredFields() works on class objects within the same module (jar file / source tree) but not outside. Let's research how this can be mitigated.

[jlink]

The embarrassing thing is that the problem has been present at least since Java 19. It's not easy to get hold of Java 18 nowadays, so I didn't try that one.
This also means that not many people seem to use nested grouping of properties. Otherwise, the problem would have shown up much earlier.

[duponter]

I was thinking the same. So I have augmented my Github workflow to run the sample project with every JDK from 17 to 21. The problem already occurs when compiling with Java 18. I am wondering what has changed that causes a different behaviour of the getDeclaredFields() method.

As we were already using JUnit 5's Nested annotation, it was a no brainer to use the Group annotation as well. Off-topic, but Intellij does not recognize a Test Class when it contains only Property tests nested in Groups. For now, we add JUnit 5 Nested annotation as well in such cases (even though we only have Property tests).