[Bug]: UI tests unexpectedly not run if CI if outside the ui package

[mcomella]

edit: see #20386 (comment) for an updated theory and explanation.

Steps to reproduce

• Run unit tests locally, specifically ManufacturersCodeTest, LocaleViewHoldersTest, or AppRequestInterceptorTest
• Run UI tests locally, specifically StartupExcessiveResourceUsageTest

Expected behaviour

Tests pass: if CI is running all the tests and correctly reporting errors then either:

• Master should have X next to the commits for the failing tests (they don't)
• Local builds should be passing (they're not)

Actual behaviour

Some tests fail (particularly the ones named)

Device name

P2

Android version

?

Firefox release type

Firefox Nightly

Firefox version

?

Device logs

No response

Additional information

I'm on commit from up/master 1bf89b6

┆Issue is synchronized with this Jira Task

[mcomella]

I looked through the logs for a PR #20176 I have written.

The test-report artifact unexpectedly only lists one UI test as having been run: org.mozilla.fenix.ui.NavigationToolbarTest. test-report.zip

The unit tests identified the test failure I left in from my PR as I expected but misattributed the failure of DefaultRecentBookmarksControllerTest to DefaultSessionControlControllerTest – strange. log.txt

[mcomella]

I have a theory that the UI tests are only running on the org.mozilla.fenix.ui package, when they should be running on all of the subpackages in org.mozilla.fenix.

To test this, I created a PR where I moved the start up test that fails locally from the omf.perf directly to omf.ui. Backing up my theory, the test failed: #20695

This implies any UI tests outside of the omf.ui directory are not running on CI. These tests appear to be:

$ ag -l "@Test" app/src/androidTest/java/org/mozilla/fenix/ --ignore ui
app/src/androidTest/java/org/mozilla/fenix/screenshots/DefaultHomeScreenTest.kt
app/src/androidTest/java/org/mozilla/fenix/screenshots/MenuScreenShotTest.kt
app/src/androidTest/java/org/mozilla/fenix/glean/BaselinePingTest.kt
app/src/androidTest/java/org/mozilla/fenix/syncintegration/SyncIntegrationTest.kt
app/src/androidTest/java/org/mozilla/fenix/perf/SampleBenchmark.kt
app/src/androidTest/java/org/mozilla/fenix/perf/StartupExcessiveResourceUseTest.kt

[mcomella]

This configuration file looks like it might be the one that defines what tests get run: https://searchfox.org/mozilla-mobile/rev/7456e758b207cd608863dba9ff1191383fd2d44f/fenix/automation/taskcluster/androidTest/flank-x86.yml#41

Currently, it's set up as an allowlist of androidTest subdirectories:

  test-targets:
   - package org.mozilla.fenix.ui
   - package org.mozilla.fenix.glean

I'm assuming there's a directory we don't want to run (screenshots?) so for future proofing, this should probably be a blocklist with the directories we don't want to run rather than an allowlist of directories we do. If there's nothing we don't want to run, then the test target should be org.mozilla.fenix.

[AaronMT]

Hi Mike,

There are a couple of paths forward here with the easiest being a change to the existing Flank configuration files in project. We use Flank-x86.yml as the configuration payload for Firebase Test Lab which is determined by script here and is selected for use on Pull Requests and Merge.

With AndroidJUnit filters like notPackage we can inverse select packages we do not wish to be ran on the daily, e.g, as you identified above screenshots and syncintegration (of which both are ran in different environments). It is just a matter of identifying which packages you wish to exclude and everything else is ran.

test-targets:
   - notPackage org.mozilla.fenix.screenshots
   - notPackage org.mozilla.fenix.syncintegration

I tried this out a WIP PR #20841 and it seemed to work (with a performance test failure; didn't really look at the reason).

Does it make sense to be running /perf/ tests on the daily here? Note: This configuration is tuned to use a Pixel2 AVD still.

[mcomella]

Does it make sense to be running /perf/ tests on the daily here? Note: This configuration is tuned to use a Pixel2 AVD still.

Currently, we're not running any tests that have long runtimes like a benchmark test and we want them to run for each PR/merge. The device configuration generally doesn't matter.

For context, the current test does runtime analysis: it starts the app once and counts things like the number of times strict mode is suppressed in order to detect if additional strict mode suppressions have been added, potentially causing perf regressions.

[mcomella]

I think this can be closed via 5ff2c21. Thanks @AaronMT !