Flaky instrumented tests

[tobrun]

Capturing from @friedbunny and @1ec5 that they have noticed flaky executions of the test suite on Firebase testlab. This issue has additional background in #3539. Note that these tests have been running reliable since #10333.

Impact

To quantify the impact, 8 out of 100 firebase executions are suffering from this issue. The first mention of the tests being flaky goes back to Thursday, 01-18-2018.

Example of a failed test

The corresponding video is showing that we weren't able to load the map. In other words, the OnMapReady callback wasn't invoked before the OnIdleResource timed out.

[tobrun]

Adding here that I'm seeing the following issue lately:

More details are available at [https://console.firebase.google.com/project/android-gl-native/testlab/histories/bh.1470d2ab45df6f1f/matrices/8101458085030057087].
+---------+-------------------------+-----------------------+
| OUTCOME |     TEST_AXIS_VALUE     |      TEST_DETAILS     |
+---------+-------------------------+-----------------------+
| Passed  | sailfish-26-en-portrait | 634 test cases passed |
+---------+-------------------------+-----------------------+
Downloading from: gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/
Copying gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/MapboxGLAndroidSDKTestApp-debug-androidTest.apk...
Skipping existing item: file://platform/android/MapboxGLAndroidSDKTestApp/build/outputs/apk/debug/MapboxGLAndroidSDKTestApp-debug.apk
Copying gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/sailfish-26-en-portrait/instrumentation.results...
Copying gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/sailfish-26-en-portrait/logcat...
Copying gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/sailfish-26-en-portrait/test_result_0.xml...
Copying gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/sailfish-26-en-portrait/video.mp4...
NotFoundException: 404 gs://test-lab-186672a0qp5bq-ycr70axads3nc/2018-01-31_12:30:44.832843_OqtD/sailfish-26-en-portrait/test_result_0.xml does not exist.
CommandException: 1 file/object could not be transferred.
Exited with code 1

While this doesn't originate from a flaky test, it does make our CI unreliable.

[mollymerp]

got another one 😒 https://circleci.com/gh/mapbox/mapbox-gl-native/67704?utm_campaign=build-failed&utm_medium=email&utm_source=notification

I don't have the permissions to look at the output in firebase.

┌─────────┬─────────────────────────┬─────────────────────────────────────────────┐
│ OUTCOME │     TEST_AXIS_VALUE     │                 TEST_DETAILS                │
├─────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ Failed  │ sailfish-26-en-portrait │ 0 test cases failed, 258 passed, 50 skipped │
└─────────┴─────────────────────────┴─────────────────────────────────────────────┘

[friedbunny]

Saw this failure on #11442:

testSanity com.mapbox.mapboxsdk.testapp.activity.gen.maplayout.LatLngBoundsForCameraActivityTest

java.lang.RuntimeException: Could not start test for LatLngBoundsForCameraActivity.
The ViewHierarchy doesn't contain a view with resource id = R.id.mapView or 
the Activity doesn't contain an instance variable with a name equal to mapboxMap.
You can resolve this issue by adding the requirements above or
 add LatLngBoundsForCameraActivity to the platform/android/scripts/exclude-activity-gen.json to blacklist the Activity from being generated.

	at com.mapbox.mapboxsdk.testapp.activity.BaseActivityTest.beforeTest(BaseActivityTest.java:49)
	at java.lang.reflect.Method.invoke(Native Method)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:76)
	at android.support.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61)
	at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:433)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

[friedbunny]

Unclear to me why, but this build on release-boba for #11462 got killed for exceeding 20 minutes.

[friedbunny]

This release-boba build for #11636 failed with:

21:21:28 Retrieving Post-test Package Stats information from the device.
21:21:28 Done. Test time = 371 (secs)
ERROR: (gcloud.firebase.test.android.run) Firebase Test Lab infrastructure failure: Exhausted post processing attempts
Exited with code 20

[friedbunny]

This release-boba build for #11637 failed on testTelemetryDialog (com.mapbox.mapboxsdk.testapp.maps.widgets.AttributionTest):

Test failed to run to completion. Reason: 'Instrumentation run failed due to 'keyDispatchingTimedOut''. Check device logcat for details

[friedbunny]

This release-boba build for #11689 failed on testSanity (com.mapbox.mapboxsdk.testapp.activity.gen.maplayout.VisibilityChangeActivityTest):

android.support.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'is displayed on the screen to the user' doesn't match the selected view.
Expected: is displayed on the screen to the user
     Got: "MapView{id=2131296431, res-name=mapView, desc=Showing a Map created with Mapbox. Scroll by dragging two fingers. Zoom by pinching two fingers., visibility=GONE, width=1080, height=1584, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@9ad986, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=5}"

[julianrex]

Appears to be identical to @mollymerp 's: https://circleci.com/gh/mapbox/mapbox-gl-native/99095

More details are available at [https://console.firebase.google.com/project/android-gl-native/testlab/histories/bh.1470d2ab45df6f1f/matrices/7291472515207452075].
┌─────────┬─────────────────────────┬─────────────────────────────────────────────┐
│ OUTCOME │     TEST_AXIS_VALUE     │                 TEST_DETAILS                │
├─────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ Failed  │ sailfish-26-en-portrait │ 2 test cases failed, 387 passed, 43 skipped │
└─────────┴─────────────────────────┴─────────────────────────────────────────────┘
Exited with code 10

If you have Firebase permissions more details are available...

[friedbunny]

This release-boba build for #11810 seemingly completed successfully, but killed itself for exceeding 20 minutes anyway.

[tobrun]

Apologies for not being able to fix/prioritize this. Just looked at the impact of the issue and seeing 4 out of 100 builds failing. I feel this is on par with stability of the other integrations. Will try to find some bandwidth to look more into this issue.