Instrumentation tests with `ANDROIDX_TEST_ORCHESTRATOR`: Parameterized tests give issues

[bddckr]

I have two issues to report when running instrumentation tests while using the Android Test Orchestrator. Feel free to tell me to split these up into two issues - happy to do so!

1. Parameterized tests aren't run/don't show up at all when executing all tests of a whole class or package.
2. When running individual parameterized test methods, the generated test names don't match that of local tests/stock JUnit 5 behavior.

I've repeatedly tested this with version 1.10.0.0 and 1.10.2.0-SNAPSHOT of the plugin.

If I disable the Android Test Orchestrator (by simply commenting out execution = "ANDROIDX_TEST_ORCHESTRATOR"), it all behaves as it should - exactly as the local tests do.

While the Wiki says to use @UseTechnicalNames, I've not needed it so far. Annotating the test class with that does not resolve the issues reported here.

Test Class

import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.EnumSource

internal class MyLocalTest {
    @ParameterizedTest
    @EnumSource
    fun test(myEnum: MyEnum) {

    }

    @ParameterizedTest(name = "custom display name {0} - works !")
    @EnumSource
    fun testWithCustomDisplayName(myEnum: MyEnum) {

    }

    enum class MyEnum {
        A, B, C
    }
}

Works: Local Test

When running the tests for that whole class as a local test, the following is displayed in Android Studio:

It also works when executing all tests for the whole package, or individual test methods.

Fails: Run Class/Package As Instrumentation Test

However, if we run that same class (renamed to MyTest in the screenshot below) or the whole package its in as an instrumentation test, we first get no results at all:

If we check logcat, we can find this:

10:44:07.116 TestRunner          I  run started: 1 tests
10:44:07.117 TestRunner          I  started: initializationError(MyTest$MyEnum)
10:44:07.117 Orchestr...istener  W  testStarted: JUnit reported MyTest$MyEnum#initializationError; discarding as bogus.
10:44:07.118 TestRunner          E  failed: initializationError(MyTest$MyEnum)
10:44:07.118 TestRunner          E  ----- begin exception -----
                                    java.lang.RuntimeException: Invalid test class 'MyTest$MyEnum': No test methods found
                                    	at androidx.test.internal.runner.EmptyTestRunner.<init>(EmptyTestRunner.java:31)
                                    	at androidx.test.internal.runner.junit4.AndroidJUnit4Builder.runnerForClass(AndroidJUnit4Builder.java:78)
                                    	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
                                    	at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
                                    	at androidx.test.internal.runner.AndroidRunnerBuilder.runnerForClass(AndroidRunnerBuilder.java:152)
                                    	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
                                    	at androidx.test.internal.runner.DirectTestLoader.doCreateRunner(DirectTestLoader.java:45)
                                    	at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:64)
                                    	at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:842)
                                    	at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:663)
                                    	at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:437)
                                    	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2361)
10:44:07.118 TestRunner          E  ----- end exception -----
10:44:07.118 Orchestr...istener  W  testFailure: JUnit reported MyTest$MyEnum#initializationError; discarding as bogus.
10:44:07.118 TestRunner          I  finished: initializationError(MyTest$MyEnum)
10:44:07.118 Orchestr...istener  W  testFinished: JUnit reported MyTest$MyEnum#initializationError; discarding as bogus.
10:44:07.120 TestRunner          I  run finished: 1 tests, 1 failed, 0 ignored

Note: We see a similar exception in logcat when we change one of the test methods to take a String together with @ValueSource. It will then say Invalid test class 'java.lang.String' instead.

Kinda Works: Wrong Names When Running Method

If we however run each test individually, it works, although the names are not as expected:

[mannodermaus]

Hey, thanks for reporting. The docs are indeed outdated, there is no point to the technical name annotation anymore (in fact, it's been deprecated by JUnit team a while back). The idea behind it is still the way to go, since the orchestrator requires the test names to be formatted in a specific way - it's just that the library didn't do it correctly.

With #342, this should be improved. It forces test names into the legacy format when the Android Instrumentation is running with the orchestrator. That means they look ugly (e.g. testWithCustomDisplayName[1] instead of custom display name 1 - works !), but at least the orchestrator works at all.

That PR also adds a validation that you cannot run orchestrator together with parallel tests. This should be available in the snapshot within 15 minutes or so. Cheers!

[bddckr]

Thank you!!!

I can confirm that it works as you describe when using plugin version 1.10.2.0-SNAPSHOT/instrumentation dependencies version 1.5.0-SNAPSHOT.