No issue: add Android Emulator action for contributor PRs

[AaronMT]

Adds a subset (those recently selected as fairly stable) to run via android-emulator-runner a HAXM enabled macOS VM. VM startup, AVD creation / boot time, and Gradle test-run for this small subset of tests (see below) takes a little less than 20 minutes. A full smoke-test run trying on personal repository took about 45 minutes. A full UI test-suite ran took just over 90 minutes. No parallelization or retries here compared to Firebase 😩

Recommendation: continue with a small subset of UI tests to at least get something running for this scenario on contributor PR.

Tests included in this draft (@sv-ohorvath selected these for beta execution recently):

• org.mozilla.fenix.ui.NavigationToolbarTest#visitURLTest
• org.mozilla.fenix.ui.HistoryTest#visitedUrlHistoryTest
• org.mozilla.fenix.ui.SmokeTest#openMainMenuSettingsItemTest
• org.mozilla.fenix.ui.SmokeTest#toggleSearchSuggestions
• org.mozilla.fenix.ui.SmokeTest#deleteCollectionTest
• org.mozilla.fenix.ui.SmokeTest#noHistoryInPrivateBrowsingTest
• org.mozilla.fenix.ui.NoNetworkAccessStartupTests#noNetworkConnectionStartupTest

Draft for feedback

[rpappalax]

Looks good so far, @AaronMT In contrast to the 20 minutes for execution time here, how long does the Fenix UI job take? Also, is that 20 minutes = build + unit test execution + UI test execution?

[AaronMT]

Looks good so far, @AaronMT In contrast to the 20 minutes for execution time here, how long does the Fenix UI job take?

On Firebase it looks like recent reporting of successful job completion (of commits under https://github.com/mozilla-mobile/fenix/commits/master) of ui-test-x86-debug are anywhere around 18-30 minutes (e.g, here's one that's 27 min https://github.com/mozilla-mobile/fenix/runs/2240724573)

Also, is that 20 minutes = build + unit test execution + UI test execution?

Yes

[AaronMT]

Re-running a few more times for posting results. Pixel 3_a. Times are total time for all checks.

• https://github.com/mozilla-mobile/fenix/actions/runs/749716825 (~25 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/749868448 (~27 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/752400935 (test failure) ❌

org.mozilla.fenix.ui.SmokeTest > toggleSearchSuggestions[test(AVD) - 9] FAILED 
974
	androidx.test.espresso.PerformException: Error performing 'single click - At Coordinates: 539, 2010 and precision: 16, 16' on view 'with id: org.mozilla.fenix.debug:id/mozac_browser_toolbar_edit_url_view'.
975
	at androidx.test.espresso.PerformException$Builder.build(PerformException.java:5)

• https://github.com/mozilla-mobile/fenix/actions/runs/752501087 (~28 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/752706417 (~28 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/756146051 (test failure) ❌ (same as above)
• https://github.com/mozilla-mobile/fenix/actions/runs/756320978 (~25 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/756320978 (test failure) ❌

org.mozilla.fenix.ui.SmokeTest > deleteCollectionTest[test(AVD) - 9] FAILED 
1009
	java.lang.AssertionError
1010
	at org.junit.Assert.fail(Assert.java:87)

• https://github.com/mozilla-mobile/fenix/actions/runs/756320978 (~30 min)
• https://github.com/mozilla-mobile/fenix/actions/runs/756762363 (~23 min)

I think for now we could start with removing those two tests. We could also limit to running only org.mozilla.fenix.ui.NavigationToolbarTest#visitURLTest. The same test that runs on nightly builds. This is the one that is battle hardened. This at least lets one know that a build successfully can start up and visit a site.

@rpappalax any preference here?

[AaronMT]

Minimizing the run to one test org.mozilla.fenix.ui.NavigationToolbarTest#visitURLTest

[rpappalax]

LGTM

[codecov-commenter]

Codecov Report

Merging #18901 (be94234) into master (b396ede) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #18901      +/-   ##
============================================
+ Coverage     34.60%   34.62%   +0.02%     
  Complexity     1623     1623              
============================================
  Files           539      539              
  Lines         21869    21876       +7     
  Branches       3273     3273              
============================================
+ Hits           7567     7575       +8     
  Misses        13403    13403              
+ Partials        899      898       -1     

Impacted Files	Coverage Δ	Complexity Δ
...rc/main/java/org/mozilla/fenix/BrowserDirection.kt	100.00% <0.00%> (ø)	2.00% <0.00%> (ø%)
...rg/mozilla/fenix/settings/account/AccountUiView.kt	36.58% <0.00%> (+2.43%)	4.00% <0.00%> (ø%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b396ede...be94234. Read the comment docs.