Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Xamarin.Android.Build.Tests] Add support for testing the UI #2855

Merged
merged 6 commits into from Apr 16, 2019

Conversation

Projects
None yet
3 participants
@dellis1972
Copy link
Contributor

dellis1972 commented Mar 20, 2019

Context #2608

This commit adds support for testing that the
app launches on device AND can be interacted
with. It does this by making use of a few features
of adb.

First is adb input. This allow us to send taps and
clicks to the device. It simulates user input. So
we can use this to say click a button.

Next up is the uiautomator this allows use to get
the complete ui layout for the active activity.
Using

exec-out uiautomator dump /dev/tty

we can send this data to the console. As it is
xml we can then parse it looking for our ui element
we want. We can then get the bounds of the control
and pass that information to adb input tap.

This commit also adds support for testing that
the timezone set on a device is correctly picked
up by mono.

And finally we add support for testing that the
debugger can actaully connect to the application.
These tests are only run if the commerial parts
of the product are available.

Seeing as all of these new tests REQUIRE the
emulator or device they have been split out into
a new MSBuildDeviceIntegration project. This
is built as part of the main solution. However
it is run as part of the make run-apk-tests
call. This is so we can be sure the emulator is
running.

In future any MSBuild related tests that REQUIRE
a device should be placed in this new project.

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch 3 times, most recently from 83d7e15 to 3da30ec Mar 27, 2019

@dellis1972 dellis1972 removed the do-not-merge label Apr 1, 2019

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch from f13fd74 to efdfb57 Apr 2, 2019

@dellis1972 dellis1972 marked this pull request as ready for review Apr 2, 2019

@dellis1972 dellis1972 requested a review from jonathanpeppers as a code owner Apr 2, 2019

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch from d8857ba to 2044732 Apr 3, 2019

@jonpryor

This comment has been minimized.

Copy link
Contributor

jonpryor commented Apr 3, 2019

Conceptual question: for this to work, there needs to be an Android device available on which to run the test and manipulate the UI.

Currently, unless I've forgotten something, Xamarin.Android.Build.Tests.dll doesn't consistently execute in an environment in which the emulator is running; the emulator is currently created as part of the RunApkTests target, which is run in parallel with the RunNUnitTests target. There is no guaranteed reliability that the emulator will -- or will not! -- be running when the local NUnit tests are run.

Perhaps we're just really lucky so far?

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch 3 times, most recently from b7b1dd7 to bc6ba03 Apr 5, 2019

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch 4 times, most recently from 0bca494 to 5e6d395 Apr 9, 2019

Show resolved Hide resolved .gitmodules Outdated

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch 2 times, most recently from 5a05ee3 to 953ebc5 Apr 10, 2019

@jonpryor

This comment has been minimized.

Copy link
Contributor

jonpryor commented Apr 11, 2019

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0 

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch from 953ebc5 to 13e6f67 Apr 11, 2019

@dellis1972

This comment has been minimized.

Copy link
Contributor Author

dellis1972 commented Apr 11, 2019

Test failures are as follows

Test Result (5 failures / -1)
SslTest.Mono.Android_Tests, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsMultiDex, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsAppBundle, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
AndroidClientHandlerTests.Mono.Android_Tests, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release
AndroidClientHandlerTests.Mono.Android_TestsAppBundle, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release
@dellis1972

This comment has been minimized.

Copy link
Contributor Author

dellis1972 commented Apr 11, 2019

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0 

That is a very very weird crash. Maybe its from the nunit runner... Its an actual test crash.

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch from a87c8db to acfd213 Apr 11, 2019

[Xamarin.Android.Build.Tests] Add support for testing the UI
This commit adds support for testing that the
app launches on device AND can be interacted
with. It does this by making use of a few features
of `adb`.

First is `adb input`. This allow us to send taps and
clicks to the device. It simulates user input. So
we can use this to say click a button.

Next up is the `uiautomator` this allows use to get
the complete ui layout for the active activity.
Using

	exec-out uiautomator dump /dev/tty

we can send this data to the console. As it is
xml we can then parse it looking for our ui element
we want. We can then get the bounds of the control
and pass that information to `adb input tap`.

This commit also adds support for testing that
the timezone set on a device is correctly picked
up by mono.

And finally we add support for testing that the
debugger can actaully connect to the application.
These tests are only run if the commerial parts
of the product are available.

Seeing as all of these new tests REQUIRE the
emulator or device they have been split out into
a new `MSBuildDeviceIntegration` project. This
is built as part of the main solution. However
it is run as part of the `make run-apk-tests`
call. This is so we can be sure the emulator is
running.

In future any MSBuild related tests that REQUIRE
a device should be placed in this new project.

@dellis1972 dellis1972 force-pushed the dellis1972:addondevicetesting branch from acfd213 to fb81f6a Apr 15, 2019

@dellis1972

This comment has been minimized.

Copy link
Contributor Author

dellis1972 commented Apr 15, 2019

@dellis1972

This comment has been minimized.

Copy link
Contributor Author

dellis1972 commented Apr 15, 2019

Commercial build failure is

 EXEC : error : option --boot-class-path not allowed with target 11

looks like that java version problem

@jonpryor

This comment has been minimized.

Copy link
Contributor

jonpryor commented Apr 16, 2019

@dellis1972 asked:

Not sure why this test failed: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/569/testReport/junit/Mono/Android_TestsMultiDex/Possible_Crash___Release/

The problem is described in commit a972624:

We have been seeing adb crash during RunApkTests:

If you download xa-test-results-v9.2.99.163_Darwin-x86_64_pr_07dd6aee-Release.zip, then bin/TestRelease/msbuild-20190415T070436-Target-RunApkTests.binlog contains:

Task "RunInstrumentationTests"
  Task Parameter:AdbTarget=-s emulator-5570
  Task Parameter:LogLevel=Verbose
  Task Parameter:PackageName=Mono.Android_TestsMultiDex
  Task Parameter:Component=Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation
  Task Parameter:NUnit2TestResultsFile=…bin/TestRelease/TestResult-Mono.Android_TestsMultiDex.xml
  Task Parameter:LogcatFilename=…/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
  Task Parameter:ToolExe=adb
  Task Parameter:ToolPath=/Users/builder/android-toolchain/sdk/platform-tools
  Task Parameter:Timeout=3600000
  Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570  shell am instrument  -e "loglevel Verbose" -w "Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation"
  Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570  logcat -v threadtime -d
    appending stdout to file: …/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
/Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/build-tools/scripts/TestApks.targets(226,5): warning : * daemon not running; starting now at tcp:5037

i.e. adb is "not running" when it most certainly should have already been running, i.e. it crashed.

Looks like bumping to platform-tools 28.0.2 didn't entirely fix things. :-(

jonpryor added some commits Apr 16, 2019

@jonpryor jonpryor merged commit bf58a88 into xamarin:master Apr 16, 2019

1 of 3 checks passed

Ubuntu PR Release Pipeline Build Build started for merge commit.
Details
macOS PR Release Pipeline Build Build started for merge commit.
Details
license/cla All CLA requirements met.
Details
@jonathanpeppers

This comment has been minimized.

Copy link
Member

jonathanpeppers commented Apr 16, 2019

Ah! we merged this and it breaks the Windows build...

external\debugger-libs\Mono.Debugging.Soft\SoftDebuggerSession.cs(2606,30): Error CS0234: The type or namespace name 'Unix' does not exist in the namespace 'Mono' (are you missing an assembly reference?)
Process 'msbuild.exe' exited with code '1'.

jonpryor added a commit that referenced this pull request Apr 16, 2019

[MSBuildDeviceIntegration] Fix build error when Cecil libs (#2985)
Commit 66d0165 [broke the macOS build][0]:

	…/xamarin-android/tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets(18,5): error MSB3030: Could not copy the file "../../bin/TestRelease/Xamarin.Android.Cecil.Mdb.dll" because it was not found.

This was probably "overlooked" because the builds for PR #2980 didn't
include bf58a88/PR #2855.

Update the paths used for `Xamarin.Android.Cecil*.dll` so that
`MSBuildDeviceIntegration.targets` can run without error.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/463/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.