Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Intermittent UI test failure - < SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest> #23816

Closed
AndiAJ opened this issue Feb 18, 2022 · 21 comments
Closed

Intermittent UI test failure - < SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest> #23816

AndiAJ opened this issue Feb 18, 2022 · 21 comments
Labels
eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer eng:ui-test UI Tests

Comments

@AndiAJ
Copy link
Collaborator

AndiAJ commented Feb 18, 2022 •
edited by data-sync-user

Firebase Test Run: Firebase link

Stacktrace:

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifySuggestedUserName(BrowserRobot.kt:628)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:299)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:292)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:292)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at androidx.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80)
at androidx.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61)
at androidx.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:531)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)

Build: 2/17 Main

┆Issue is synchronized with this Jira Task
@AndiAJ AndiAJ added eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer eng:ui-test UI Tests labels Feb 18, 2022
@github-actions github-actions bot added the needs:triage Issue needs triage label Feb 18, 2022
@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Feb 22, 2022

Flaky1x on 2/21 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyPrefilledLoginCredentials(BrowserRobot.kt:635)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:302)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:292)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:292)
at java.lang.reflect.Method.invoke(Native Method)

@sv-ohorvath sv-ohorvath removed the needs:triage Issue needs triage label Feb 25, 2022
@sv-ohorvath
Copy link
Contributor

Flaky on 2/25 Main: link

ava.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464) at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:294) at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:292) at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112) at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:292)

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 3, 2022

Flaky 1x on 3/2 Main Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 3, 2022

Flaky 1x on 3/3 Main Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 10, 2022

Flaky 1x on 3/9 Main Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 22, 2022

Flaky 1x on 3/21 Main Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 23, 2022

Failed 1x on 3/22 Main
Firebase link
Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Mar 24, 2022 •
edited

Failed 2x on 3/23 Main
Firebase link
Firebase link
Firebase link
Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Apr 7, 2022

Flaky 1x on 4/6 Main Firebase link

@sv-ohorvath
Copy link
Contributor

Flaky on debug 14/4: link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented Apr 28, 2022

Flaky 1x on 4/27 Main Firebase link

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 2, 2022

Flaky 1x on 4/30 Main Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference
at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:312)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:310)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:310)

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 2, 2022

Flaky 1x on 5/1 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference
at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:312)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:310)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:310)

@Mugurell
Copy link
Contributor

Mugurell commented May 2, 2022

This was also seen in #25051.

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 2, 2022

Failed on #24903
Firebase link
Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference
at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:312)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:310)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:310)

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 3, 2022 •
edited

Flaky 3x on 5/3 Main
Firebase link
Firebase link
Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference
at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:312)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:310)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:310)

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 4, 2022

Flaky 1x on 5/4 Main Firebase link
java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.test.uiautomator.UiObject2.click()' on a null object reference
at org.mozilla.fenix.ui.robots.BrowserRobot.saveLoginFromPrompt(BrowserRobot.kt:464)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:312)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyMultipleLoginsSelectionsTest$2.invoke(SettingsPrivacyTest.kt:310)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:112)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyMultipleLoginsSelectionsTest(SettingsPrivacyTest.kt:310)

@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 5, 2022

Flaky 1x on 5/5 Main Firebase link

@mcarare mcarare mentioned this issue May 5, 2022
Merged
3 tasks
@mcarare
Copy link
Contributor

mcarare commented May 5, 2022

Another fail on a PR: https://console.firebase.google.com/u/1/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/6341292660255641491/executions/bs.15b8ef5984786818/testcases/1/test-cases

Amejia481 added a commit to badboy/fenix that referenced this issue May 5, 2022
@Amejia481
Disable intermittent tests mozilla-mobile#24843 and mozilla-mobile#23816
768d007
Amejia481 added a commit to badboy/fenix that referenced this issue May 5, 2022
@Amejia481
Disable intermittent tests mozilla-mobile#24843 and mozilla-mobile#23816
2b0580a
mergify bot pushed a commit that referenced this issue May 5, 2022
@Amejia481 @mergify
Disable intermittent tests #24843 and #23816
7b3574d
@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 6, 2022

Disabled in #25060

@AndiAJ AndiAJ added the eng:disabled-test For any test that has been disabled label May 6, 2022
AndiAJ added a commit to AndiAJ/fenix that referenced this issue May 6, 2022
@AndiAJ
For mozilla-mobile#23816 fix disabled verifyMultipleLoginsSelectionsT…
f259e2a 
…est UI test
AndiAJ added a commit to AndiAJ/fenix that referenced this issue May 9, 2022
@AndiAJ
For mozilla-mobile#23816 fix disabled verifyMultipleLoginsSelectionsT…
df3b46b 
…est UI test
mergify bot pushed a commit that referenced this issue May 9, 2022
@AndiAJ @mergify
For #23816 fix disabled verifyMultipleLoginsSelectionsTest UI test
98455c0
@AndiAJ AndiAJ removed the eng:disabled-test For any test that has been disabled label May 9, 2022
@AndiAJ
Copy link
Collaborator Author

AndiAJ commented May 9, 2022

Fixed in #25105

@AndiAJ AndiAJ closed this as completed May 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
eng:intermittent-test Evil test failures that only happen now-and-then and are a productivity killer eng:ui-test UI Tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sv-ohorvath @Mugurell @mcarare @AndiAJ