Add Test Orchestrator support.

[artem-zinnatullin]

You can trick Composer to work with Orchestrator right now, but more official solution would be better.

Also, as we've measured in #66 instrumenting each test separately can add 3-5 seconds overhead for each test which can be unacceptable for those who run them frequently (each PR/commit), but we need to actually measure this with Orchestrator.

See:

• https://android-developers.googleblog.com/2017/07/android-testing-support-library-10-is.html
• https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator

[artem-zinnatullin]

@ming13 any additions here since you've tried Orchestrator? I am also curious about test run duration in compare to regular am instrumentation.

[arturdryomov]

Okay, I’m doing everything manually from the command line.

$ adb install -r app.apk
$ adb install -r test.apk

$ adb install -r orchestrator.apk
$ adb install -r test-services.apk

Just instrument

adb shell "am instrument -r -w \
  -e package com.gojuno.actually.some.package \
  com.gojuno.rider.test/android.support.test.runner.JunoAndroidRunner"

• There is instrument output.
• Final result — OK (21 tests), INSTRUMENTATION_CODE: -1.
• Time — 5 runs in 760 seconds.

Orchestrator

adb shell 'CLASSPATH=$(pm path android.support.test.services) app_process / \
  android.support.test.services.shellexecutor.ShellMain am instrument -r -w \
  -e package com.gojuno.actually.some.package \
  -e targetInstrumentation com.gojuno.rider.test/android.support.test.runner.JunoAndroidRunner \
  android.support.test.orchestrator/.AndroidTestOrchestrator'

• There is no instrument output.
  • The only thing gets printed after finished run — INSTRUMENTATION_RESULT: stream=.
• Final result — OK (0 tests), INSTRUMENTATION_CODE: 0.
  • Please notice that tests were executed just fine.
• Time — 5 runs in 591 seconds (SURPRISE).

BTW.

-e numShards and -e shardIndex runner arguments aren't currently supported

Links:

• https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator
• https://developer.android.com/topic/libraries/testing-support-library/release-notes.html

[arturdryomov]

I’ll file the issue about missing instrument output (maybe) tomorrow.

[artem-zinnatullin]

Very interesting to see that it took THAT less time. I suspect that not all tests were run

[artem-zinnatullin]

Great, please crosslink this issue with instrumentation output issue in Google bugtracker!

[arturdryomov]

Very interesting to see that it took THAT less time. I suspect that not all tests were run

I actually counted them visually and they all were here.

[arturdryomov]

Here we go — https://issuetracker.google.com/issues/64486480

[artem-zinnatullin]

You probably better copy "steps to reproduce" there as well

[handstandsam]

@artem-zinnatullin - You mentioned there is a way to currently trick composer into using the test orchestrator? What is that trick?

I have been able to run my tests with the orchestrator via adb shell like @ming13 did, but haven't figured out how to do it via Composer. Thanks!

[artem-zinnatullin]

Unfortunately since adb shell am command is really different with Orchestrator there is no way to "trick" Composer to use Orchestrator, also it's unclear from https://issuetracker.google.com/issues/64486480 and release notes if they fixed instrumentation output of Orchestrator or not.

Once they fix it, we can start implementing support for Orchestrator :)

[yunikkk]

I've just tried the latest version of test libs and discovered that it writes xml reports to the sdcard /sdcard/odo/*, eg.

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="6" failures="0" errors="0" skipped="1" time="25.982" timestamp="2017-10-25T23:22:20" hostname="localhost">
  <properties />
  <testcase name="testAboutAndCheckUpdates" classname="com.gojuno.driver.robotPattern.tests.menu.about.AboutUITest" start-time="1508973716.203" end-time="1508973718.141" time="1.938" />
  <testcase name="testCommonHelp" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForAuthorizedUserUITest" start-time="1508973720.870" end-time="1508973723.102" time="2.232" />
  <testcase name="testMoreOptions" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForAuthorizedUserUITest" start-time="1508973725.621" end-time="1508973728.267" time="2.646" />
  <testcase name="testHelpContactDataAreDisplayed" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForUnauthorizedUserUITest" start-time="1508973730.770" end-time="1508973731.869" time="1.099" />
  <testcase name="testErrorOnCheckForUpdatesIsDisplayed" classname="com.gojuno.driver.robotPattern.tests.unauthorized.LoginWithOldAppVersionTest" start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9">
    <skipped />
  </testcase>
  <testcase name="testADSHardBlockScreenIsDisplayed" classname="com.gojuno.driver.robotPattern.tests.unauthorized.LoginWithOldAppVersionTest" start-time="1508973736.861" end-time="1508973739.655" time="2.794" />
</testsuite>

[artem-zinnatullin]

Well we can hook into that, but that won't be "reactive", instrumentation output is printed asap, but I bet JUnit report only appears after test run

start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9"

lol wut

[yunikkk]

Yup, reports appears only after run, checked that.
Also, sharding seems to be working, too.

[yunikkk]

start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9"
its for ignored test =)

[originx]

any news on this issue, is it possible to implement it with current version of orchestrator.

Btw have you ever considered JUnit listeners, or piggybacking on existing ones:
https://android.googlesource.com/platform/frameworks/testing/+/3352c37/androidtestlib/src/com/android/test/runner/listener/

?

[yunikkk]

Well, I haven't tried since October, will need to recheck status of issues we faced.

Regarding listeners - to use them we'll need to run tests with our own customized TestRunner, I guess. That will broke cases for the users who implements their own runners in the test apk. Not sure if there is a workaround...
@artem-zinnatullin wdyt about listeners & runner?

[originx]

you can add the listeners to the existing test runner:
To specify one or more RunListeners to observe the test run: -e listener com.foo.Listener,com.foo.Listener2
https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner.html

there is also a possibility to add them in manifest or via gradle configuration ie. instrumentation arguments.

or in the end you can go like spoon did and provide the customized testrunner via the gradle plugin:
https://github.com/square/spoon/tree/master/spoon-runner/src/main/java/com/squareup/spoon

but I think it is a safer way to go compared to the bash output from the instrumentation process?

[trevjonez]

even if they use the listener interface they still need to communicate back to the host machine for report generation. parsing instrumentation process output is tedious and perhaps fragile from the maintenance perspective but the next obvious communication medium for streaming the data back to the host would be some sort of network connection from the device which can probably be statistically proven as a large source of flake if any data was tracked/analyzed.

perhaps if not a network connection the listener would write to a file on device that could then be tailed over adb on the host?

Other than being able to control the communication format I don't see a huge value to doing it that way.

[artem-zinnatullin]

Yay, I don't see benefit in using listeners right now. Orchestrator needs to be reinvestigated, last time we checked it didn't produce correct instrumentation output. Alternative could be to have own "Orchestrator", but I'd prefer not to do this. Another alternative could be to start test run that runs single test, stops the app and clears its date. Do it for every test and you have Orchestrator. It can be done on top of existing logic, we'll just need to parse apk to get test classes and methods. The only problem with this solution is that it'll work well only with JUnit based tests.

…

On Thu, Mar 8, 2018, 8:33 AM Trevor Jones ***@***.***> wrote: even if they use the listener interface they still need to communicate back to the host machine for report generation. parsing instrumentation process output is tedious and perhaps fragile from the maintenance perspective but the next obvious communication medium for streaming the data back to the host would be some sort of network connection from the device which can probably be statistically proven as a large source of flake if any data was tracked/analyzed. perhaps if not a network connection the listener would write to a file on device that could then be tailed over adb on the host? Other than being able to control the communication format I don't see a huge value to doing it that way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#90 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7B3KlR1i-OimAZwd-g1lEwauWwi2XDks5tcV1jgaJpZM4Ok3Va> .

[evgengal]

Any news on this issue?

[artem-zinnatullin]

Last related PR: #138

However Google is working on Nitrogen that overlaps with Composer functionality which might affect our plans

[evgengal]

Do you mean Nitrogen OS?

[dsvoronin]

Nope, "Project nitrogen" announced on IO https://www.youtube.com/watch?v=wYMIadv9iF8

[CristianGM]

@artem-zinnatullin If the year ends without Google releasing Nitrogen ( it wouldn't be the first time, thats why usually they don't say dates ) all devs using Orchestrator will have wasted a year without Composer ( so, having a good sharding and report ).

I don't get the answer saying Nitrogen might affect Composer plans, is like abandoning the project just because eventually there will be something better.