ActivityScenarioRule incompatible with release build type

[softartdev]

Description

After migrate from deprecated ActivityTestRule to ActivityScenarioRule ui-tests with release build type failed:

java.lang.AssertionError: Activity never becomes requested state "[STARTED, CREATED, DESTROYED, RESUMED]" (last lifecycle transition = "PRE_ON_CREATE")
at androidx.test.core.app.ActivityScenario.waitForActivityToBecomeAnyOf(ActivityScenario.java:338)

Steps to Reproduce

The build type that test against is controlled by the testBuildType flag. Switch between release & debug by setting property from CLI.
Use keeper library for keep test APK sources on minified release build.
Replace all ActivityTestRule with activityScenarioRule.

Actual Results

Bug reproduces on my pet-project's GitHub-Actions CI:
https://github.com/softartdev/NoteDelight/runs/2433814114?check_suite_focus=true#step:7:295
Note that step for debug build type ui-tests was successful:
https://github.com/softartdev/NoteDelight/runs/2433814114?check_suite_focus=true#step:6:1062

Expected Results

Both release & debug build types ui-tests was successful before test-rules update:
https://github.com/softartdev/NoteDelight/runs/2433726325?check_suite_focus=true#step:7:302
https://github.com/softartdev/NoteDelight/runs/2433726325?check_suite_focus=true#step:6:1082
Most test results are flaky, more stable in SDK 26~29 API versions.

AndroidX Test and Android OS Versions

AndroidX Test libs version 1.3.0
targetSdkVersion 30

Link to a public git repo demonstrating the problem:

https://github.com/softartdev/NoteDelight/tree/scenario

[adazh]

Mind also posting your Logcat logs?

[softartdev]

Release builds don't print log by default. If set debuggable true, then one of tests (CreateRemoveNoteTest) application logcat will following:

2021-04-27 02:06:28.963 20933-20933/? I/artdev.noteroo: Not late-enabling -Xcheck:jni (already on)
2021-04-27 02:06:28.974 20933-20933/? E/artdev.noteroo: Unknown bits set in runtime_flags: 0x8000
2021-04-27 02:06:28.976 20933-20933/? W/artdev.noteroo: Unexpected CPU variant for X86 using defaults: x86
2021-04-27 02:06:29.005 20933-20933/com.softartdev.noteroom W/ActivityThread: Package uses different ABI(s) than its instrumentation: package[com.softartdev.noteroom]: x86, null instrumentation[com.softartdev.noteroom.test]: null, null
2021-04-27 02:06:29.010 20933-20933/com.softartdev.noteroom I/artdev.noteroo: The ClassLoaderContext is a special shared library.
2021-04-27 02:06:29.263 20933-20933/com.softartdev.noteroom I/chatty: uid=10154(com.softartdev.noteroom) identical 2 lines
2021-04-27 02:06:29.263 20933-20933/com.softartdev.noteroom I/artdev.noteroo: The ClassLoaderContext is a special shared library.
2021-04-27 02:06:29.490 20933-20933/com.softartdev.noteroom W/artdev.noteroo: Accessing hidden method Landroid/app/Instrumentation;->execStartActivity(Landroid/content/Context;Landroid/os/IBinder;Landroid/os/IBinder;Landroid/app/Activity;Landroid/content/Intent;ILandroid/os/Bundle;)Landroid/app/Instrumentation$ActivityResult; (greylist, linking, allowed)
2021-04-27 02:06:29.491 20933-20933/com.softartdev.noteroom W/artdev.noteroo: Accessing hidden method Landroid/app/Instrumentation;->execStartActivity(Landroid/content/Context;Landroid/os/IBinder;Landroid/os/IBinder;Ljava/lang/String;Landroid/content/Intent;ILandroid/os/Bundle;)Landroid/app/Instrumentation$ActivityResult; (greylist, linking, allowed)
2021-04-27 02:06:29.497 20933-20933/com.softartdev.noteroom I/MultiDex: VM with version 2.1.0 has multidex support
2021-04-27 02:06:29.497 20933-20933/com.softartdev.noteroom I/MultiDex: Installing application
2021-04-27 02:06:29.497 20933-20933/com.softartdev.noteroom I/MultiDex: VM has multidex support, MultiDex support library is disabled.
2021-04-27 02:06:29.507 20933-20933/com.softartdev.noteroom W/ComponentDiscovery: Class com.google.firebase.dynamicloading.DynamicLoadingRegistrar is not an found.
2021-04-27 02:06:29.514 20933-20933/com.softartdev.noteroom I/FirebaseApp: Device unlocked: initializing all Firebase APIs for app [DEFAULT]
2021-04-27 02:06:29.528 20933-20965/com.softartdev.noteroom W/artdev.noteroo: Unsupported class loader
2021-04-27 02:06:29.533 20933-20933/com.softartdev.noteroom I/FirebaseCrashlytics: Initializing Firebase Crashlytics 17.4.1
2021-04-27 02:06:29.533 20933-20965/com.softartdev.noteroom I/DynamiteModule: Considering local module com.google.android.gms.measurement.dynamite:48 and remote module com.google.android.gms.measurement.dynamite:21
2021-04-27 02:06:29.533 20933-20965/com.softartdev.noteroom I/DynamiteModule: Selected local version of com.google.android.gms.measurement.dynamite
2021-04-27 02:06:29.543 20933-20933/com.softartdev.noteroom W/FA: Failed to register event listener on calling thread. Trying again on the dynamite thread.
2021-04-27 02:06:29.615 20933-20933/com.softartdev.noteroom I/FirebaseInitProvider: FirebaseApp initialization successful
2021-04-27 02:06:29.634 20933-20933/com.softartdev.noteroom I/MonitoringInstr: Instrumentation started!
2021-04-27 02:06:29.653 20933-20977/com.softartdev.noteroom D/NetworkSecurityConfig: No Network Security Config specified, using platform default
2021-04-27 02:06:29.871 20933-20969/com.softartdev.noteroom V/FA: App measurement collection enabled
2021-04-27 02:06:29.872 20933-20969/com.softartdev.noteroom V/FA: App measurement enabled for app package, google app id: com.softartdev.noteroom, 1:936732740110:android:c93ab423c78ce920
2021-04-27 02:06:29.874 20933-20969/com.softartdev.noteroom I/FA: App measurement initialized, version: 39065
2021-04-27 02:06:29.874 20933-20969/com.softartdev.noteroom I/FA: To enable debug logging run: adb shell setprop log.tag.FA VERBOSE
2021-04-27 02:06:29.874 20933-20969/com.softartdev.noteroom I/FA: To enable faster debug mode event logging run:
      adb shell setprop debug.firebase.analytics.app com.softartdev.noteroom
2021-04-27 02:06:29.874 20933-20969/com.softartdev.noteroom D/FA: Debug-level message logging enabled
2021-04-27 02:06:29.878 20933-20969/com.softartdev.noteroom V/FA: Persisting first open: 1619471189520
2021-04-27 02:06:29.885 20933-20969/com.softartdev.noteroom V/FA: App measurement setting deferred collection: false
2021-04-27 02:06:29.974 20933-20933/com.softartdev.noteroom I/[Koin]: [init] declare Android Context
2021-04-27 02:06:29.981 20933-20969/com.softartdev.noteroom V/FA: Checking service availability
2021-04-27 02:06:29.986 20933-20969/com.softartdev.noteroom V/FA: Service available
2021-04-27 02:06:29.991 20933-20969/com.softartdev.noteroom V/FA: Connecting to remote service
2021-04-27 02:06:29.999 20933-20933/com.softartdev.noteroom I/[Koin]: loaded 14 definitions - 2.0502 ms
2021-04-27 02:06:29.999 20933-20933/com.softartdev.noteroom D/[Koin]: instances started in 0.04804 ms
2021-04-27 02:06:30.000 20933-20933/com.softartdev.noteroom D/[Koin]: +- 'com.softartdev.notedelight.util.PreferencesHelper'
2021-04-27 02:06:30.000 20933-20933/com.softartdev.noteroom D/[Koin]: | create instance for [Single:'com.softartdev.notedelight.util.PreferencesHelper']
2021-04-27 02:06:30.001 20933-20933/com.softartdev.noteroom D/[Koin]: +- 'android.content.Context'
2021-04-27 02:06:30.001 20933-20933/com.softartdev.noteroom D/[Koin]: | create instance for [Single:'android.content.Context',binds:android.app.Application]
2021-04-27 02:06:30.002 20933-20933/com.softartdev.noteroom D/[Koin]: |- 'android.content.Context' in 0.43175 ms
2021-04-27 02:06:30.004 20933-20933/com.softartdev.noteroom D/[Koin]: |- 'com.softartdev.notedelight.util.PreferencesHelper' in 3.7317 ms
2021-04-27 02:06:30.010 20933-20933/com.softartdev.noteroom I/OrchestrationListener: OrchestrationListener connected to service
2021-04-27 02:06:30.011 20933-20933/com.softartdev.noteroom I/MonitoringInstr: No JSBridge.
2021-04-27 02:06:30.013 20933-20969/com.softartdev.noteroom V/FA: Connection attempt already in progress
2021-04-27 02:06:30.019 20933-20969/com.softartdev.noteroom V/FA: Connection attempt already in progress
2021-04-27 02:06:30.020 20933-20969/com.softartdev.noteroom D/FA: Connected to remote service
2021-04-27 02:06:30.020 20933-20969/com.softartdev.noteroom V/FA: Processing queued up service tasks: 3
2021-04-27 02:06:30.031 20933-20983/com.softartdev.noteroom I/UsageTrackerFacilitator: Usage tracking disabled
2021-04-27 02:06:30.034 20933-20983/com.softartdev.noteroom W/artdev.noteroo: Unable to resolve Lcom/softartdev/notedelight/ui/CreateRemoveNoteTest; annotation class 3258
2021-04-27 02:06:30.035 20933-20983/com.softartdev.noteroom I/chatty: uid=10154(com.softartdev.noteroom) roidJUnitRunner identical 2 lines
2021-04-27 02:06:30.045 20933-20983/com.softartdev.noteroom W/artdev.noteroo: Unable to resolve Lcom/softartdev/notedelight/ui/CreateRemoveNoteTest; annotation class 3258
2021-04-27 02:06:30.049 20933-20983/com.softartdev.noteroom I/artdev.noteroo: Unable to resolve java.lang.Class<com.softartdev.notedelight.ui.CreateRemoveNoteTest> annotation class 3258
2021-04-27 02:06:30.056 20933-20983/com.softartdev.noteroom D/TestExecutor: Adding listener androidx.test.internal.runner.listener.LogRunListener
2021-04-27 02:06:30.057 20933-20983/com.softartdev.noteroom D/TestExecutor: Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
2021-04-27 02:06:30.057 20933-20983/com.softartdev.noteroom D/TestExecutor: Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
2021-04-27 02:06:30.057 20933-20983/com.softartdev.noteroom D/TestExecutor: Adding listener leakcanary.FailTestOnLeakRunListener
2021-04-27 02:06:30.057 20933-20983/com.softartdev.noteroom W/artdev.noteroo: Unable to resolve Lleakcanary/FailTestOnLeakRunListener; annotation class 3258
2021-04-27 02:06:30.057 20933-20983/com.softartdev.noteroom I/artdev.noteroo: Unable to resolve java.lang.Class<com.softartdev.notedelight.ui.CreateRemoveNoteTest> annotation class 3258
2021-04-27 02:06:30.058 20933-20983/com.softartdev.noteroom W/artdev.noteroo: Unable to resolve Lcom/softartdev/notedelight/ui/CreateRemoveNoteTest; annotation class 3258
2021-04-27 02:06:30.059 20933-20983/com.softartdev.noteroom I/chatty: uid=10154(com.softartdev.noteroom) roidJUnitRunner identical 6 lines
2021-04-27 02:06:30.059 20933-20983/com.softartdev.noteroom W/artdev.noteroo: Unable to resolve Lcom/softartdev/notedelight/ui/CreateRemoveNoteTest; annotation class 3258
2021-04-27 02:06:30.059 20933-20983/com.softartdev.noteroom I/artdev.noteroo: Unable to resolve java.lang.Class<com.softartdev.notedelight.ui.CreateRemoveNoteTest> annotation class 3258
2021-04-27 02:06:30.060 20933-20983/com.softartdev.noteroom I/TestRunner: run started: 1 tests
2021-04-27 02:06:30.065 20933-20983/com.softartdev.noteroom I/artdev.noteroo: Unable to resolve java.lang.Class<com.softartdev.notedelight.ui.CreateRemoveNoteTest> annotation class 3258
2021-04-27 02:06:30.066 20933-20983/com.softartdev.noteroom I/artdev.noteroo: Unable to resolve java.lang.Class<com.softartdev.notedelight.ui.CreateRemoveNoteTest> annotation class 3258
2021-04-27 02:06:30.068 20933-20983/com.softartdev.noteroom I/TestRunner: started: createRemove(com.softartdev.notedelight.ui.CreateRemoveNoteTest)
2021-04-27 02:06:30.078 20933-20983/com.softartdev.noteroom W/Settings: Setting always_finish_activities has moved from android.provider.Settings.System to android.provider.Settings.Global, returning read-only value.
2021-04-27 02:06:31.451 20933-20946/com.softartdev.noteroom I/artdev.noteroo: Background concurrent copying GC freed 27701(2740KB) AllocSpace objects, 24(992KB) LOS objects, 49% free, 1733KB/3466KB, paused 372us total 452.473ms
2021-04-27 02:06:35.293 20933-20969/com.softartdev.noteroom V/FA: Inactivity, disconnecting from the service
2021-04-27 02:07:15.122 20933-20983/com.softartdev.noteroom E/TestRunner: failed: createRemove(com.softartdev.notedelight.ui.CreateRemoveNoteTest)
2021-04-27 02:07:15.122 20933-20983/com.softartdev.noteroom E/TestRunner: ----- begin exception -----
2021-04-27 02:07:15.123 20933-20983/com.softartdev.noteroom E/TestRunner: java.lang.AssertionError: Activity never becomes requested state "[CREATED, RESUMED, STARTED, DESTROYED]" (last lifecycle transition = "PRE_ON_CREATE")
        at androidx.test.core.app.ActivityScenario.waitForActivityToBecomeAnyOf(ActivityScenario.java:338)
        at androidx.test.core.app.ActivityScenario.launchInternal(ActivityScenario.java:272)
        at androidx.test.core.app.ActivityScenario.launch(ActivityScenario.java:207)
        at androidx.test.ext.junit.rules.ActivityScenarioRule.lambda$new$1$ActivityScenarioRule(ActivityScenarioRule.java:79)
        at androidx.test.ext.junit.rules.ActivityScenarioRule$$Lambda$1.get(Unknown Source:4)
        at androidx.test.ext.junit.rules.ActivityScenarioRule.before(ActivityScenarioRule.java:103)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at androidx.test.ext.junit.runners.AndroidJUnit4.run(AndroidJUnit4.java:154)
        at org.junit.runners.Suite.runChild(Suite.java:128)
        at org.junit.runners.Suite.runChild(Suite.java:27)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
        at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
2021-04-27 02:07:15.123 20933-20983/com.softartdev.noteroom E/TestRunner: ----- end exception -----
2021-04-27 02:07:15.123 20933-20983/com.softartdev.noteroom D/OrchestrationListener: Sending TestFailure event: Activity never becomes requested state "[CREATED, RESUMED, STARTED, DESTROYED]" (last lifecycle transition = "PRE_ON_CREATE")
2021-04-27 02:07:15.128 20933-20983/com.softartdev.noteroom I/TestRunner: finished: createRemove(com.softartdev.notedelight.ui.CreateRemoveNoteTest)
2021-04-27 02:07:15.136 20933-20983/com.softartdev.noteroom I/TestRunner: run finished: 1 tests, 1 failed, 0 ignored
2021-04-27 02:07:15.139 20933-20983/com.softartdev.noteroom I/MonitoringInstr: waitForActivitiesToComplete() took: 0ms

[softartdev]

Exactly the same error in another project:
https://github.com/softartdev/ConwaysGameOfLife/runs/2650593289?check_suite_focus=true#step:7:268

[brozikcz]

We have a similar issue with these errors.

The workaround which works for us is to change androidTestImplementation to debugImplementation.

[softartdev]

@brozikcz, I don't think how these debug- statements can solve this RELEASE build type issue.

[fbcbl]

@brozikcz which are the dependencies you are suggesting to change to debugImplementation? and why is that the solution?

[brozikcz]

debugImplementation androidx.test:core:1.4.0 works for us, what it is exactly wrong I don't have idea but this works

[softartdev]

@brozikcz debug-builds works for us as well, but this issue about release-builds

[PGrube26]

I have the same problem and wanted to try your workaround. In a small project it's working as expected but in our multi module app with many dependencies even this workaround is not working and the test fails because of the described error message.

UPDATE: I saw that we are deleting the intent data in our activity, after I removed this part my activity got started correctly.

[top-master]

Fixed

The real fix was to downgrade, like:

• classpath 'com.android.tools.build:gradle:7.1.3' (in build.gradle file)

In other words, stay far away from latest-Gradle-plugin versions,
and always test for yourself to see if it's falsely-labeled-stable or is really-stable.

Some people seem to use debugImplementation as workaround,
which's called workaround for a reason,
as that does not fix the root cause.

Background

• We did update to Gradle 7.5.1 to support JDK 18,

• but suddenly many activity-tests were failing for no reason,

• then after 2 days testing, we noticed that downgrading AndroidGradle-plugin (or whatever com.android.tools.build:gradle is called) was enough

  No need to downgrade Gradle-itself (the .zip file).

  else, we would in gradle-wrapper.properties file do:

  distributionUrl=https\://services.gradle.org/distributions/gradle-7.2-bin.zip
  

  (as we use that version since months without issue).