Avoid execution of tests before generating the test image

[melix]

Proposal

Currently, tests are executed in JVM mode, so that we can generate a list of test ids, which are in turn used as an input to generate the native test image. However, technically speaking, we don't need to run the tests: we only need to collect the test ids. So we could have some kind of "dry run" mode for tests which actually generate the test id list, but do not execute the tests in practice.

Note that this is orthogonal to this JUnit issue which basically integrates the feature of generating the test id list into JUnit: in addition, we also need a way to skip the execution altogether.

It is very well possible that such a mechanism already exists in JUnit, because for example Gradle's test distribution does something similar, with an initial discovery phase. @marcphilipp might have some insights.

[sbrannen]

Currently, tests are executed in JVM mode, so that we can generate a list of test ids, which are in turn used as an input to generate the native test image. However, technically speaking, we don't need to run the tests: we only need to collect the test ids.

That's correct: we do not need to execute the tests.

But... collecting the list of UIDs is not the only goal of the TestExecutionListener approach. One of the main goals achieved by using the listener when running the test suite on the JVM is that we are using the user's custom configuration (e.g., from the test task in Gradle) to select the tests, test naming pattern, included/excluded test engines, included/excluded tags, etc.

So we could have some kind of "dry run" mode for tests which actually generate the test id list, but do not execute the tests in practice.

The Feature actually makes use of a "dry run" to register test classes for reflection:

native-build-tools/common/junit-platform-native/src/main/java/org/graalvm/junit/platform/JUnitPlatformFeature.java

Lines 145 to 163 in b20363e

	private TestPlan registerTestPlan(Launcher launcher, List<Path> classpath) {
	LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
	.selectors(getSelectors(classpath))
	.build();
	TestPlan testPlan = launcher.discover(request);
	testPlan.getRoots().stream()
	.flatMap(rootIdentifier -> testPlan.getDescendants(rootIdentifier).stream())
	.map(TestIdentifier::getSource)
	.filter(Optional::isPresent)
	.map(Optional::get)
	.filter(ClassSource.class::isInstance)
	.map(ClassSource.class::cast)
	.map(ClassSource::getJavaClass)
	.forEach(this::registerTestClassForReflection);
	return testPlan;
	}

The same thing (i.e., "discovery" without "execution") can be done to find the UIDs of tests.

But... we shouldn't do that blindly. Rather, if we go that route I think we should come up with a mechanism to copy the user's test configuration from the test task (taking Gradle as an example again) to reuse that configuration when building up the LauncherDiscoveryRequest.

[dzou]

I've been doing research on getting more complex unit tests that use mocking libraries working with native-build-tools, and I think there might be one benefit to running the tests beforehand.

I noticed libraries like Mockito or EasyMock use cglib for creating mock objects, and they will internally call ClassLoader.defineClass(..) to dynamically create mock classes for unit tests.

This is referred to as "dynamic class loading" and is not supported in GraalVM by default:

     Caused by: java.lang.reflect.InvocationTargetException
       java.lang.reflect.Method.invoke(Method.java:566)
       org.easymock.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:459)
       org.easymock.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:339)
       [...]
     Caused by: com.oracle.svm.core.jdk.UnsupportedFeatureError: Defining classes from new bytecodes run time.
       com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:87)
       java.lang.ClassLoader.defineClass(ClassLoader.java:425)
       [...]

However, there looks like some workaround solutions being attempted; the solution sounds like it extends the native-image-agent to produce some class definition files for GraalVM at runtime in a folder used by the compiler at run time.

So if you run the tests beforehand in JVM mode, you can attach the native-image-agent to that run and record these dynamic class loads done by the mocking libraries using that feature. Then theoretically the work done in JVM mode can be carried over for the native-image run. There's probably other blockers but this might keep the door open to getting unit tests that use mocking libraries working.

[fniephaus]

Is this ticket still relevant, @melix?

[melix]

Yes it is.