Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests/ci] Add BCL Test job to Azure pipelines #3420

Merged
merged 4 commits into from Aug 9, 2019

Conversation

@pjcollins
Copy link
Member

pjcollins commented Jul 31, 2019

Adds a new job to build and run the BCL Test Suite on an emulator.

@pjcollins pjcollins force-pushed the pjcollins:bcltest-azp branch 2 times, most recently from 90a43a6 to 012ba52 Aug 1, 2019
@pjcollins pjcollins marked this pull request as ready for review Aug 2, 2019
@pjcollins pjcollins requested review from grendello and jonpryor as code owners Aug 2, 2019
@pjcollins

This comment has been minimized.

Copy link
Member Author

pjcollins commented Aug 2, 2019

The "Run Test Suite Foo" functionality isn't incorporated in azure-pipelines.yaml, but it's very helpful for when you want to run a specific xUnit test or set of tests locally.

@pjcollins

This comment has been minimized.

Copy link
Member Author

pjcollins commented Aug 2, 2019

We're hitting a failure in MonoTests.System.TimeZoneTest.TestCtors now that we're running this suite on the Azure hosted macOS pool as those machines appear to be set to GMT (therefore our emulator is set to GMT). I think this is an ignorable failure - mono/mono#15987.

@pjcollins pjcollins force-pushed the pjcollins:bcltest-azp branch from 523ee25 to aa2059f Aug 6, 2019
@pjcollins pjcollins force-pushed the pjcollins:bcltest-azp branch from aa2059f to fc075d6 Aug 6, 2019
@pjcollins

This comment has been minimized.

Copy link
Member Author

pjcollins commented Aug 7, 2019

We're only attempting to pull xUnit test results from the emulator once in the entire build log, yet somehow the Jenkins Release Pipeline Build is back up to ~357k test results...

[2019-08-06T23:01:46.546Z] Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb pull "/storage/emulated/0/Android/data/Xamarin.Android.Bcl_Tests/files/Documents/TestResults.xUnit.xml" "/Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/tests/../bin/TestRelease/TestResult-Xamarin.Android.Bcl_Tests.xunit.xml"

Perhaps rebuilds result in double result publishing on Jenkins? This is this case with Azure Pipelines, as a few stages were re-ran against the latest commit and the initial results which were published were not overwritten, but instead appended to.

Jenkins weirdness aside, I think we're ready to merge this. All failures in the latest AZP run are known issues, known intermittent issues, or caused by an attempt to re-publish an artifact due to a stage being re-ran (which will be fixed in an upcoming PR).

@jonpryor jonpryor merged commit 3d9b154 into xamarin:master Aug 9, 2019
8 of 14 checks passed
8 of 14 checks passed
Ubuntu PR Release Pipeline Build Build finished. No test results found.
Details
Xamarin.Android Build #merge-fdf17a8237ecd5f0114f8c0242aa30152a6dc3b2-1 had test failures
Details
Xamarin.Android (Test BCL With Emulator) Test BCL With Emulator failed
Details
Xamarin.Android (Test MSBuild Mac) Test MSBuild Mac failed
Details
Xamarin.Android (Windows Build and Test) Windows Build and Test failed
Details
macOS PR Release Pipeline Build Build finished. 357504 tests run, 1992 skipped, 8 failed.
Details
Xamarin.Android (Mac Build) Mac Build succeeded
Details
Xamarin.Android (Prepare bundle) Prepare bundle succeeded
Details
Xamarin.Android (Test APK Instrumentation) Test APK Instrumentation succeeded
Details
Xamarin.Android (Test Designer Mac) Test Designer Mac succeeded
Details
Xamarin.Android (Test Designer Windows) Test Designer Windows succeeded
Details
Xamarin.Android (Test MSBuild With Emulator Mac) Test MSBuild With Emulator Mac succeeded
Details
Xamarin.Android (Test TimeZoneInfo With Emulator Mac) Test TimeZoneInfo With Emulator Mac succeeded
Details
license/cla All CLA requirements met.
Details
@pjcollins pjcollins deleted the pjcollins:bcltest-azp branch Aug 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.