Run UI tests nightly def

[isabelrios]

This PR continues the work to have the UI tests running against the correct and each build, beta, nightly or debug...

Pull Request checklist

• Tests: This PR includes thorough tests or an explanation of why it does not
• Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
• Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

To download an APK when reviewing a PR:

1. click on Show All Checks,
2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
3. click on the "Fenix - assemble" task, then click "Run Artifacts".
4. the APK links should be on the left side of the screen, named for each CPU architecture

[isabelrios]

This PR continues the work done in PR #17017, it got a first review from Johan and the approach was discussed with him. The idea is to run at least one test on the Nightly build in addition to the tests that run in the debug build so that if it fails we know something is not correct with the Nightly build that is used by the cron job to uplift it to the playstore.

There will be a new task added ui-test-x86-nightly (see test running on Firebase) running one test to start with: visitURLTest.

The UI test on Nightly will be run as the existing UI tests, per push in master.

[isabelrios]

This file will be removed, just added to see that the ui-test-x86-nightly task is created successfully

[escapewindow]

I ran taskgraph-diff against this to see what it changes in the taskgraph. The diff output is here.

Overall this looks good. I do see the on-push task, the nightly task, and the signing task. I think the other diffs are due to the difficulty of diffing large json blobs against each other.

Thanks!

[isabelrios]

@st3fan just to let you know about this and hear your feedback if you think this is useful or not and in case we need to/is good to add some mergify rules... Thanks!

[rpappalax]

LGTM

[rpappalax]

do we want 50 here?

[isabelrios]

I don't think so 😓 It would be enough with one since we are running only one test atm. According to the comments, that could be -1:

echo "To run UI tests on ARM device shard (1 test / shard)"
echo "$ ui-test.sh arm64-v8a -1"

[codecov-io]

Codecov Report

Merging #17513 (f5b2e9d) into master (c8c0131) will increase coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #17513      +/-   ##
============================================
+ Coverage     32.50%   32.56%   +0.06%     
+ Complexity     1320     1314       -6     
============================================
  Files           458      455       -3     
  Lines         19227    19089     -138     
  Branches       2698     2669      -29     
============================================
- Hits           6249     6217      -32     
+ Misses        12463    12382      -81     
+ Partials        515      490      -25     

Impacted Files	Coverage Δ	Complexity Δ
...main/java/org/mozilla/fenix/TelemetryMiddleware.kt	79.41% <0.00%> (-7.08%)	18.00% <0.00%> (-1.00%)
...zilla/fenix/components/metrics/MetricController.kt	33.33% <0.00%> (-4.27%)	0.00% <0.00%> (ø%)
...c/main/java/org/mozilla/fenix/home/HomeFragment.kt	0.00% <0.00%> (-2.47%)	0.00% <0.00%> (-5.00%)
...java/org/mozilla/fenix/components/metrics/Event.kt	33.96% <0.00%> (-0.81%)	0.00% <0.00%> (ø%)
.../src/main/java/org/mozilla/fenix/utils/Settings.kt	71.90% <0.00%> (-0.18%)	36.00% <0.00%> (ø%)
...rc/main/java/org/mozilla/fenix/FenixApplication.kt	7.05% <0.00%> (-0.13%)	5.00% <0.00%> (ø%)
...a/org/mozilla/fenix/browser/BaseBrowserFragment.kt	11.26% <0.00%> (-0.13%)	19.00% <0.00%> (-1.00%)
app/src/main/java/org/mozilla/fenix/ext/Nimbus.kt	0.00% <0.00%> (ø)	0.00% <0.00%> (ø%)
...pp/src/main/java/org/mozilla/fenix/FeatureFlags.kt	100.00% <0.00%> (ø)	4.00% <0.00%> (ø%)
...ain/java/org/mozilla/fenix/components/Analytics.kt	60.37% <0.00%> (ø)	1.00% <0.00%> (ø%)
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8c0131...79850a3. Read the comment docs.