[UITests] BrowserRobot verifyPageContent() fails on random tests

[sv-ohorvath]

• verifyPageContent() --
  closePrivateTabTest
  https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/4711066025457837200/executions/bs.388b4b88ce9c05b/testcases/2/test-cases

testDownloadPrompt
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/7572548415764500855/executions/bs.25bdda78baa6975a/testcases/1/test-cases

tabMediaControlButtonTest
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/6271613602422171703/executions/bs.c9b1d2c1d1d6ab5b/testcases/2/test-cases

audioPlaybackSystemNotificationTest
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/6271613602422171703/executions/bs.79bc2a17f53e73f0/test-cases

• enterURLAndEnterToBrowser() -- [fixed for now]
  Navigation Toolbar goForwardTest
  https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5442255196437551623/executions/bs.9b8cd54c0f4a93bd/testcases/2/test-cases

goBackTest:
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5442255196437551623/executions/bs.72ee6d576d88bfe1/testcases/2/test-cases

and more

┆Issue is synchronized with this Jira Task

[sv-ohorvath]

enterURLAndEnterToBrowser() was failing because the notification shade was open covering the app :| Will be fixed in #10649

[sv-ohorvath]

verifyPageContent() was fixed by 32e74aa

[sv-ohorvath]

Still seeing tests fail at verifyPageContent():
openNewPrivateTabTest
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/8010288854433600147/executions/bs.a0e73
4097ce18e79/test-cases

verifyRemoveTopSite
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5429184379757340395/executions/bs.b0383f5dd57fc239/test-cases

deleteMultipleSelectionTest
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5429184379757340395/executions/bs.1a20f54849080d09/testcases/1/test-cases

[TejaswiKarasani]

Hi @sv-ohorvath and @isabelrios ,

I fear this is the same case for my test failing in the PR raised:

#10822

https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/7774504630902755015

[isabelrios]

@TejaswiKarasani I may be wrong but I don't think is the same issue. I understood from the test and the video of the failure that it is failing on Homscreen trying to open the menu. I don't see that open in the video.. Once you fix that robot transition commented on the PR, let's see if it keeps failing and where...thanks!

[TejaswiKarasani]

@TejaswiKarasani I may be wrong but I don't think is the same issue. I understood from the test and the video of the failure that it is failing on Homscreen trying to open the menu. I don't see that open in the video.. Once you fix that robot transition commented on the PR, let's see if it keeps failing and where...thanks!

Yeah I will fix that and re-test it again🙌.

[sv-ohorvath]

@AaronMT I can't reproduce locally any of these failing tests. Do you have any suggestions on how to prevent it from failing? Or, do you think we need a developer looking into it?

[AaronMT]

Yes, ideally we should be using SessionLoadedIdlingResource but these pages load so fast from local server that the timing of the idling resource count is not ideal

https://github.com/mozilla-mobile/fenix/blob/bc11bfae96b90a639aba3dccf34a766f41cf7df3/app/src/androidTest/java/org/mozilla/fenix/helpers/SessionLoadedIdlingResource.kt

[AaronMT]

I also think it might be simple as By.textContains instead of text() call, I have a change in my draft PR #10928

[AaronMT]

This still seems to be problematic despite the change. Some thoughts in my head (I don't know enough about UI Automator about why this function is hitting a timeout so often):

Here's the problematic function

fun verifyPageContent(expectedText: String) {
        val mDevice = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
        mDevice.waitNotNull(Until.findObject(By.textContains(expectedText)), waitingTime)
    }

Depending on what the test is doing can this declaration of UIDevice get the wrong instance? Should mDevice be defined earlier? It seems like mDevice is never not null so perhaps something is going awry here. Why this function in particular?

Should we just kill this function entirely?

[AaronMT]

My only guess is that the layout tree at the time of the waitNotNull tree check does not reflect the current layout tree at the time of idle page content. You can see this in the video, the page is fully loaded but the waitNotNull check still fails. The snapshot at the time is wrong. I don't know how to proceed here. I'm in favor of removing the function.

[rpappalax]

@AaronMT OK, fair enough. @AaronMT @sv-ohorvath So if we remove the verify content, do we have an alternative verify or if we skip it, does that weaken the validity of the test?

[sv-ohorvath]

Disabled for now the most frequent failures:
HistoryTest: deleteMultipleSelectionTest, openHistoryInPrivateTabTest
TabbedBrowsingTest: openNewPrivateTabTest, openNewTabTest
NavToolbarTest: refreshPageTest

[sv-ohorvath]

Fixed in #12499 I think this can be closed.