Can't run any test after Bump androidx.test:runner from 1.4.0 to 1.5.1.

[DenBond7]

Description

Hello. After updating androidx.test:runner from 1.4.0 to 1.5.1. I can't run any test - not via Gradle not via Android Studio. It seems like runner can't find tests. Please look at the picture

Steps to Reproduce

1. Download https://github.com/FlowCrypt/flowcrypt-android
2. Switch to dependabot/gradle/androidx.test-runner-1.5.1 branch
3. run tests via ./gradlew :FlowCrypt:connectedDevTestDebugAndroidTest

Expected Results

Gradle should run tests

Actual Results

I see starting 0 tests on emulator

AndroidX Test and Android OS Versions

Android 12L (API level 32)

Link to a public git repo demonstrating the problem:

More details can be found here https://github.com/FlowCrypt/flowcrypt-android/pull/2064/files

[brettchabot]

Does logcat show an error ? I'm trying to repro based on your instructions but the gradle command just seems to hang

[DenBond7]

I'm trying to repro based on your instructions but the gradle command just seems to hang

Yup, that's what I saw too.

Does logcat show an error ?

@brettchabot Thank you for the hint. I've found the following in logcat

2022-11-11 09:15:06.009 5877-5877/com.flowcrypt.email.debug I/MonitoringInstr: Instrumentation started!
2022-11-11 09:15:06.014 5877-5877/com.flowcrypt.email.debug D/AndroidRuntime: Shutting down VM
    
    --------- beginning of crash
2022-11-11 09:15:06.014 5877-5877/com.flowcrypt.email.debug E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.flowcrypt.email.debug, PID: 5877
    java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/test/platform/io/PlatformTestStorageRegistry;
        at androidx.test.internal.runner.RunnerArgs$Builder.<init>(RunnerArgs.java:248)
        at androidx.test.runner.AndroidJUnitRunner.parseRunnerArgs(AndroidJUnitRunner.java:393)
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:302)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6726)
        at android.app.ActivityThread.access$1500(ActivityThread.java:256)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2090)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.app.ActivityThread.main(ActivityThread.java:7842)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)
     Caused by: java.lang.ClassNotFoundException: Didn't find class "androidx.test.platform.io.PlatformTestStorageRegistry" on path: DexPathList[[zip file "/system/framework/android.test.runner.jar", zip file "/system/framework/android.test.mock.jar", zip file "/system/framework/android.test.base.jar", zip file "/data/app/~~JGTlQHmnKsflzT0MNRyuug==/com.flowcrypt.email.debug.test-k4tCgwyhE9bFZg1OxjOduQ==/base.apk", zip file "/data/app/~~uJ4tSpAd7JxSt3Y1dnD5AA==/com.flowcrypt.email.debug-ZVF8qO1HJass9CCVAYwjZg==/base.apk"],nativeLibraryDirectories=[/data/app/~~JGTlQHmnKsflzT0MNRyuug==/com.flowcrypt.email.debug.test-k4tCgwyhE9bFZg1OxjOduQ==/lib/x86_64, /data/app/~~uJ4tSpAd7JxSt3Y1dnD5AA==/com.flowcrypt.email.debug-ZVF8qO1HJass9CCVAYwjZg==/lib/x86_64, /system/lib64, /system_ext/lib64]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:218)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
        at androidx.test.internal.runner.RunnerArgs$Builder.<init>(RunnerArgs.java:248) 
        at androidx.test.runner.AndroidJUnitRunner.parseRunnerArgs(AndroidJUnitRunner.java:393) 
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:302) 
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6726) 
        at android.app.ActivityThread.access$1500(ActivityThread.java:256) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2090) 
        at android.os.Handler.dispatchMessage(Handler.java:106) 
        at android.os.Looper.loopOnce(Looper.java:201) 
        at android.os.Looper.loop(Looper.java:288) 
        at android.app.ActivityThread.main(ActivityThread.java:7842) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)

It looks like a dependency conflict.

The app uses androidx.test:monitor:1.4.0 but should androidx.test:monitor:1.6.0

Unfortunately, adding androidTestImplementation 'androidx.test:monitor:1.6.0' doesn't resolve the issue. But implementation 'androidx.test:monitor:1.6.0' fixed it and I can run tests.

[DenBond7]

You shouldn't be shipping test code in prod, looking at your repo, recent commits and PRs I think the problem is that Dependabot is bumping everything one by one.

Agree. But I can't find settings to set up Dependabot in such a way. For now, Dependabot makes PR for every dependency separately. I'm going to ask Dependabot team about such an ability.

[DenBond7]

ref dependabot/dependabot-core#6108

[TWiStErRob]

@brettchabot given the state of affairs, how about an AndroidX Test BOM? https://developer.android.com/jetpack/compose/setup#using-the-bom

[brettchabot]

It is true that the best tested path is for all androidx.test dependencies to be the latest version, but you absolutely should not have to do this. A bunch of API enforcement infrastructure was put in place in this release to detect versioning incompatibilities, and while there is more work to do on this front, I don't agree that dependabot upgrading versions in isolation is the root cause of the issue.

The root cause is adding test dependencies to 'implementation' and gradle's inability to deal properly with versioning issues resulting from that. I'd suggest either restructuring the code to avoid androidx.test dependencies in 'implementation', or configure gradle to just build a single apk for tests, just like is done for library projects. Again IMO just building all dependencies into a single apk should be the default behavior. @yuuki3655 do you remember the gradle option to do this?