Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11 #743

mateuszkwiecinski · 2020-09-15T19:35:00Z

Description

Similar to #355, but the execution hangs

Steps to Reproduce

Create new project using AS wizard
Enable Android orchestrator following the https://developer.android.com/training/testing/junit-runner
Run tests on Android Emulator with API 30

Expected Results

The test passes

Actual Results

The test hangs
logcat shows:

2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.myapplication, PID: 12791
    java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.myapplication.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
     Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.connect(OrchestratedInstrumentationListener.java:93)
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:309)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6704)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) 
        at android.os.Handler.dispatchMessage(Handler.java:106) 
        at android.os.Looper.loop(Looper.java:223) 
        at android.app.ActivityThread.main(ActivityThread.java:7656) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) 
2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidJUnitRunner: An unhandled exception was thrown by the app.
2020-09-15 21:28:39.493 12791-12791/com.example.myapplication E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash

(might be unrelated, but doesn' happen on api 29) my private app shows in addition: mkdir: ‘/storage/emulated/0/Android/data/my.app.package’: Permission denied which points at DeviceProviderInstrumentTestTask and getAdditionalTestOutputEnabled which tries to save some log file. disabling the android.enableAdditionalTestOutput property removes the mkdir error but the test still hangs

AndroidX Test and Android OS Versions

    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test:runner:1.3.0'
    androidTestUtil 'androidx.test:orchestrator:1.3.0'

Android OS Versions: the issue exists only on API 30. The test passes on lower apis.

Link to a public git repo demonstrating the problem:

https://github.com/mateuszkwiecinski/orchestrator_doesnt_work
task: ./gradlew connectedDebugAndroidTest hangs and never ends

Github action run confirming the issue: https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/1/checks?check_run_id=1119649523

The text was updated successfully, but these errors were encountered:

prfarlow1 · 2020-09-16T15:09:23Z

I am also experiencing this bug. I also made a test application that experiences the same indefinite hanging on Android 11: https://github.com/prfarlow1/UiTestApplication

LmKupke · 2020-09-21T18:31:01Z

Same issue is occurring when compiling and targeting SDK 30 and running against an emulator with 30.

2020-09-21 14:26:44.942 31419-31419/com.scompany.consumer E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.scompany.consumer, PID: 31419
java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo
java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
at android.app.ActivityThread.access$1300(ActivityThread.java:237)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:223)
at android.app.ActivityThread.main(ActivityThread.java:7656)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService

no issue when compiling and targeting 30 and running on an emulator with 29.
no issue when compiling and targeting 29 and running on an emulator with 30.

prfarlow1 · 2020-09-22T15:15:04Z

Can we get an update from the Android test team when this will be addressed? This is a really bad bug. We rely on the orchestrator for our tests to pass and right now we just can't run our tests on Android 11 test devices.

nimeacuerdo · 2020-09-28T08:59:20Z

Same here. Everything was working fine with emulator api 29 and Orchestrator until we migrated to api 30.

brettchabot · 2020-10-05T18:36:21Z

Hey all, sorry for delayed response. We are looking into this, but the fix is proving more difficult to get released than anticipated. We hope to have something out soon to address this.

tillchen · 2020-10-15T08:21:20Z

Same here. Switched to Android 10 and it's working fine for me.

brettchabot · 2020-10-15T20:16:26Z

Please give 1.3.1-alpha01 a try. This should hopefully be fixed.
See https://github.com/android/android-test/releases/tag/androidx-test-1.3.1-alpha01

mateuszkwiecinski · 2020-10-19T09:13:08Z

Can't confirm it works :/ It doesn't hang indeed, but still fails with a message:

Task :app:connectedDebugAndroidTest
Starting 0 tests on test(AVD) - 11
Tests on test(AVD) - 11 failed: Instrumentation run failed due to 'Process crashed.'

com.android.build.gradle.internal.testing.ConnectedDevice > No tests found.[test(AVD) - 11] FAILED
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack @test annotations).

Failed run:
https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/3/checks?check_run_id=1274143298

It works on devices with api <30

prfarlow1 · 2020-10-19T13:33:05Z

I also could not get my tests to work with the new 1.3.1-alpha01

brettchabot · 2020-10-19T17:10:21Z

It looks like there is an incompatibility with the testInstrumentationRunnerArguments clearPackageData: 'true' setting. I was able to get @mateuszkwiecinski sample working by commenting out that line.

Enable clearPackageData

brettchabot · 2020-10-19T20:31:42Z

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

prfarlow1 · 2020-10-19T20:55:44Z

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

What does android:forceQueryable="true" actually do? I can't find any documentation for it

brettchabot · 2020-10-19T20:58:23Z

Check out the docs on the new package visibility restrictions on API 30 here
https://developer.android.com/training/basics/intents/package-visibility

IIUC using forceQueryable basically lets other apps (like the orchestrator app) communicate with the test package.

This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator. Fixes #743 PiperOrigin-RevId: 337944463

brettchabot · 2020-10-20T18:14:36Z

Please give 1.3.1-alpha02 a try

mateuszkwiecinski · 2020-10-20T19:33:20Z

Thank you @brettchabot 👍
I can confirm it works in my sample project 🚀

brettchabot · 2020-10-20T19:40:05Z

whew thanks for confirming @mateuszkwiecinski

tadfisher · 2020-10-22T18:44:51Z

android:forceQueryable isn't recognized by AAPT2 4.1.0 and fails the build:

Execution failed for task ':Mobile-Android:processDevDebugAndroidTestResources'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > Android resource linking failed
     /home/tad/simple/mobile/Mobile-Android/Mobile-Android/build/intermediates/packaged_manifests/devDebugAndroidTest/AndroidManifest.xml:22: AAPT: error: attribute android:forceQueryable not found.

brettchabot · 2021-03-17T17:58:06Z

are you using studio 4.2 beta 6 ?

QAutomatron · 2021-03-26T16:28:46Z

Using 1.4.0-alpha05 testx and Studio 4.2b6, getting "No tests to run", most probably something crashed during init phase on Android 11 emulator.

logcat said about some new permission issue:

2021-03-26 18:21:05.133 9240-9263/androidx.test.orchestrator E/OrchestrationXmlTestRunListener: Failed to generate report data
    java.io.IOException: Operation not permitted
        at java.io.UnixFileSystem.createFileExclusively0(Native Method)
        at java.io.UnixFileSystem.createFileExclusively(UnixFileSystem.java:317)
        at java.io.File.createTempFile(File.java:2018)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.getResultFile(OrchestrationXmlTestRunListener.java:2)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.createOutputResultStream(OrchestrationXmlTestRunListener.java:8)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.generateDocument(OrchestrationXmlTestRunListener.java:3)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.orchestrationRunFinished(OrchestrationXmlTestRunListener.java:4)
        at androidx.test.orchestrator.AndroidTestOrchestrator.finish(AndroidTestOrchestrator.java:1)
        at androidx.test.orchestrator.AndroidTestOrchestrator.executeNextTest(AndroidTestOrchestrator.java:2)
        at androidx.test.orchestrator.AndroidTestOrchestrator.runFinished(AndroidTestOrchestrator.java:10)
        at androidx.test.orchestrator.TestRunnable.run(TestRunnable.java:13)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

I don't know what else to try.

btseytlinTCP · 2021-04-30T14:33:03Z

FYI 1.4.0-alpha06 released which has a fix for the forceQueryable bug: https://github.com/android/android-test/releases/tag/androidx-test-1.4.0-alpha06

krisjwolff · 2021-04-30T16:21:40Z

Still getting Cannot connect to androidx.test.orchestrator.OrchestratorService using alpha06 on Pixel 5 API 30.

If I explicitly install with --force-queryable as described on Feb 18, it works fine.

brettchabot · 2021-04-30T16:26:43Z

@krisjwolff what Studio + AGP version are you using?

You need to use Android Studio 4.2+ . Only in this version and later will Studio use the --force-queryable flag when installing the service apks.

alex-tyro · 2021-05-03T07:32:35Z

I am using 1.4.0-alpha06 and this still doesn't work for me on firebase test lab. Every other api level is fine except 11.
Firebase fails with: Cannot connect to androidx.test.orchestrator.OrchestratorService

brettchabot · 2021-05-03T16:37:25Z

@alex-tyro thanks for the report. I was able to repro on FTL and I filed bug against them. Unfortunately due to the android11 package visibility changes there is nothing we can do on the androidx.test library to address this. FTL will need the same fix as Studio: adding a '--force-queryable' flag when installing the service apks.

krisjwolff · 2021-05-03T17:25:31Z

@brettchabot Thank you for filing a bug with FTL. Is that ticket publicly-available, so I can watch it?

brettchabot · 2021-05-03T17:58:14Z

Unfortunately I could not find an easy way to file an publically visible bug. I can update this issue when FTL responds if that works?

prfarlow1 · 2021-05-03T19:47:43Z

Unfortunately I could not find an easy way to file an publically visible bug. I can update this issue when FTL responds if that works?

That would be fantastic, thank you for keeping us up-to-date.

krisjwolff · 2021-05-03T20:05:52Z

Unfortunately I could not find an easy way to file an publically visible bug. I can update this issue when FTL responds if that works?

Yes please. That's very helpful, thanks!

nimeacuerdo · 2021-05-04T06:50:14Z

For the record, these are the versions we ended up using with Android 11 emulators and so far so good:

    espresso_version = '3.4.0-alpha02'
    androidx_test_version = '1.3.1-alpha02'

with version 30.5.4 of emulator and version 9 of system-images;android-30;google_apis;x86.

brettchabot · 2021-05-04T17:55:19Z

I'd be cautious about drawing any conclusions based on Android 11 emulators. IIUC the images are dated and do not have the final package visibility handling. In my testing I saw different behavior on Android 11 devices and Android S emulator images.

larryng · 2021-05-05T16:40:54Z

I'm still seeing the issue even with installing using --force-queryable:

2021-05-05 12:36:46.126 14893-14959/? W/Binder: Caught a RuntimeException from the binder stub implementation.
    java.lang.SecurityException: Calling from not trusted UID!
        at android.app.UiAutomationConnection.throwIfCalledByNotTrustedUidLocked(UiAutomationConnection.java:525)
        at android.app.UiAutomationConnection.shutdown(UiAutomationConnection.java:429)
        at android.app.IUiAutomationConnection$Stub.onTransact(IUiAutomationConnection.java:390)
        at android.os.Binder.execTransactInternal(Binder.java:1159)
        at android.os.Binder.execTransact(Binder.java:1123)
...
2021-05-05 12:36:46.663 3626-8001/? E/MediaProvider: insertFileIfNecessary failed
    java.lang.IllegalArgumentException: Primary directory odo not allowed for content://media/external_primary/file; allowed directories are [Download, Documents]
        at com.android.providers.media.MediaProvider.ensureFileColumns(MediaProvider.java:2923)
        at com.android.providers.media.MediaProvider.ensureUniqueFileColumns(MediaProvider.java:2588)
        at com.android.providers.media.MediaProvider.insertFile(MediaProvider.java:3282)
        at com.android.providers.media.MediaProvider.insertInternal(MediaProvider.java:3826)
        at com.android.providers.media.MediaProvider.insert(MediaProvider.java:3537)
        at com.android.providers.media.MediaProvider.insertFileForFuse(MediaProvider.java:7187)
        at com.android.providers.media.MediaProvider.insertFileIfNecessaryForFuse(MediaProvider.java:7281)
2021-05-05 12:36:46.666 14866-14892/androidx.test.orchestrator E/OrchestrationXmlTestRunListener: Failed to generate report data
    java.io.IOException: Operation not permitted
        at java.io.UnixFileSystem.createFileExclusively0(Native Method)
        at java.io.UnixFileSystem.createFileExclusively(UnixFileSystem.java:317)
        at java.io.File.createTempFile(File.java:2018)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.getResultFile(OrchestrationXmlTestRunListener.java:2)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.createOutputResultStream(OrchestrationXmlTestRunListener.java:8)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.generateDocument(OrchestrationXmlTestRunListener.java:3)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.orchestrationRunFinished(OrchestrationXmlTestRunListener.java:4)
        at androidx.test.orchestrator.AndroidTestOrchestrator.finish(AndroidTestOrchestrator.java:1)
        at androidx.test.orchestrator.AndroidTestOrchestrator.executeNextTest(AndroidTestOrchestrator.java:2)
        at androidx.test.orchestrator.AndroidTestOrchestrator.runFinished(AndroidTestOrchestrator.java:10)
        at androidx.test.orchestrator.TestRunnable.run(TestRunnable.java:13)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

This is with 1.4.0-alpha06 on a Pixel 3 Android 11 (RQ1A.210205.004).

brettchabot · 2021-05-05T18:03:40Z

@larryng that seems like a separate issue than the one discussed here. Can you file a new issue?

brettchabot · 2021-05-06T01:04:16Z

@krisjwolff @prfarlow1 FTL has made a fix. They expect it should get deployed in the next week or two.

larryng · 2021-05-06T14:29:28Z

@brettchabot Apologies, the issue was with allure-kotlin, though the error messages in Orchestrator are a noisy red herring.

QAutomatron · 2021-05-06T14:31:56Z

@brettchabot Apologies, the issue was with allure-kotlin, though the error messages in Orchestrator are a noisy red herring.

have you managed to find a workaround? Can't make it work with allure-kotlin

larryng · 2021-05-06T15:12:04Z

@QAutomatron Here you go. If you have any questions, please ask there instead. I don't want to continue pinging others on this thread.

ashok07m · 2021-05-31T13:01:15Z

I am also using testInstrumentationRunnerArguments clearPackageData: 'true'
Just update core and runner versions to make it working on android 11:
androidTestImplementation 'androidx.test:core-ktx:1.4.0-alpha04'
androidTestImplementation 'androidx.test:runner:1.4.0-alpha04'
androidTestUtil 'androidx.test:orchestrator::1.4.0-alpha04'
This solved the issue for me i.e the core,runner,rules,orchestrator test deps should be updated to same version.
now all AndroidTest cases runs in queue manner and independent of each other in my android device.

vidushi2207 · 2021-06-10T05:18:10Z

I am observing the same issue with
androidTestUtil 'androidx.test:orchestrator:1.4.0-beta02'
androidTestImplementation 'androidx.test:rules:1.4.0-beta02'
androidTestImplementation 'androidx.test:runner:1.4.0-beta02'
and adding android:forceQueryable="true"

fdspbsteam · 2022-06-17T13:30:23Z

I am observing the same issue with

androidTestUtil 'androidx.test:orchestrator:1.4.1' and alpha 4 as well
androidTestImplementation 'androidx.test:rules:1.4.0
    androidTestImplementation 'androidx.test:core:1.4.0'
    // AndroidJUnitRunner and JUnit Rules
    androidTestImplementation 'androidx.test:runner:1.4.0'
    androidTestImplementation 'androidx.test:rules:1.4.0'
    
and adding android:forceQueryable="true"

this issue is hap aping in case command run via ./gradlew connectedCheck

yogurtearl · 2022-12-15T20:57:55Z

related issues for docs update: https://issuetracker.google.com/issues/262661481

mateuszkwiecinski mentioned this issue Sep 15, 2020

Add failing workflow mateuszkwiecinski/orchestrator_doesnt_work#1

Merged

brettchabot closed this as completed Oct 15, 2020

mateuszkwiecinski mentioned this issue Oct 19, 2020

Update build.gradle mateuszkwiecinski/orchestrator_doesnt_work#3

Closed

brettchabot reopened this Oct 19, 2020

brettchabot added a commit to android/testing-samples that referenced this issue Oct 19, 2020

Repro for android/android-test#743

2adf169

Enable clearPackageData

brettchabot self-assigned this Oct 19, 2020

copybara-service bot mentioned this issue Oct 19, 2020

Add forceQueryable=true to androidx.test.runner. #777

Merged

copybara-service bot closed this as completed in 1faf8cd Oct 20, 2020

copybara-service bot closed this as completed in #777 Oct 20, 2020

mateuszkwiecinski mentioned this issue Oct 20, 2020

Update build.gradle mateuszkwiecinski/orchestrator_doesnt_work#5

Merged

denrase mentioned this issue Mar 30, 2021

Integration testing for the sentry-native new unwinder getsentry/sentry-java#1359

Closed

marandaneto mentioned this issue Apr 2, 2021

Integration testing for the sentry-native new unwinder getsentry/sentry-java#1371

Closed

4 tasks

plastiv mentioned this issue May 31, 2021

Orchestrator fix for running tests on Android 11 (sdk 30) trevjonez/composer-gradle-plugin#86

Open

brettchabot mentioned this issue Jun 18, 2021

Instrumented tests don't work on API 30 using Gradle command #986

Closed

AndiAJ mentioned this issue Aug 11, 2021

connectedFocusDebugAndroidTest - No tests found mozilla-mobile/focus-android#5086

Closed

yogurtearl mentioned this issue Sep 30, 2021

Exception with using TestStorage #1141

Closed

Gelassen mentioned this issue Jun 30, 2023

[choir] Merge with the master branch Gelassen/swap#20

Closed

Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11 #743

Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11 #743

Comments

mateuszkwiecinski commented Sep 15, 2020 • edited Loading

Description

Steps to Reproduce

Expected Results

Actual Results

AndroidX Test and Android OS Versions

Link to a public git repo demonstrating the problem:

prfarlow1 commented Sep 16, 2020

LmKupke commented Sep 21, 2020 • edited Loading

prfarlow1 commented Sep 22, 2020

nimeacuerdo commented Sep 28, 2020

brettchabot commented Oct 5, 2020

tillchen commented Oct 15, 2020

brettchabot commented Oct 15, 2020

mateuszkwiecinski commented Oct 19, 2020 • edited Loading

prfarlow1 commented Oct 19, 2020

brettchabot commented Oct 19, 2020

brettchabot commented Oct 19, 2020 • edited Loading

prfarlow1 commented Oct 19, 2020

brettchabot commented Oct 19, 2020

brettchabot commented Oct 20, 2020

mateuszkwiecinski commented Oct 20, 2020

brettchabot commented Oct 20, 2020

tadfisher commented Oct 22, 2020

brettchabot commented Mar 17, 2021

QAutomatron commented Mar 26, 2021

btseytlinTCP commented Apr 30, 2021

krisjwolff commented Apr 30, 2021

brettchabot commented Apr 30, 2021

alex-tyro commented May 3, 2021

brettchabot commented May 3, 2021

krisjwolff commented May 3, 2021

brettchabot commented May 3, 2021

prfarlow1 commented May 3, 2021

krisjwolff commented May 3, 2021

nimeacuerdo commented May 4, 2021 • edited Loading

brettchabot commented May 4, 2021

larryng commented May 5, 2021

brettchabot commented May 5, 2021

brettchabot commented May 6, 2021

larryng commented May 6, 2021

QAutomatron commented May 6, 2021 • edited Loading

larryng commented May 6, 2021

ashok07m commented May 31, 2021

vidushi2207 commented Jun 10, 2021

fdspbsteam commented Jun 17, 2022

yogurtearl commented Dec 15, 2022

mateuszkwiecinski commented Sep 15, 2020 •

edited

Loading

LmKupke commented Sep 21, 2020 •

edited

Loading

mateuszkwiecinski commented Oct 19, 2020 •

edited

Loading

brettchabot commented Oct 19, 2020 •

edited

Loading

nimeacuerdo commented May 4, 2021 •

edited

Loading

QAutomatron commented May 6, 2021 •

edited

Loading