Skip to content

Fix flaky iPad UI Test - testWPcomLoginLogout#21200

Merged
jostnes merged 2 commits intotrunkfrom
fix-flaky-ipad-appSettings-cell
Jul 31, 2023
Merged

Fix flaky iPad UI Test - testWPcomLoginLogout#21200
jostnes merged 2 commits intotrunkfrom
fix-flaky-ipad-appSettings-cell

Conversation

@jostnes
Copy link
Copy Markdown
Contributor

@jostnes jostnes commented Jul 28, 2023

Description

The testWPcomLoginLogout test is flaky on iPad test runs. On iPhone test runs, it has a 100% passing rate.

When looking at the difference between iPad and iPhone, I found that on iPad, the MeTabScreen is a modal on top of the MySiteScreen. Because there is often a bit of a timing issue in CI, when the test is verifying that MeTabScreen is loaded, it's still testing against MySiteScreen element tree causing the test to hang and fail.

I'm not 100% sure this will be the right solution because, in theory, the test without any changes should work, but that isn't the reality. I'd like to try this out and monitor for a bit if it works. There are also other iPad-only flakiness that could be because of the same issue. If this works, we can apply the same to those tests too.

Testing

testWPcomLoginLogout is not flaky on iPad (with error: Failed to determine hittability of "appSettings" Cell: Activation point invalid and no suggested hit points based on element frame) and CI is 🟢

@jostnes jostnes added the Testing Unit and UI Tests and Tooling label Jul 28, 2023
@jostnes jostnes added this to the 23.0 milestone Jul 28, 2023
@wpmobilebot
Copy link
Copy Markdown
Contributor

wpmobilebot commented Jul 28, 2023

WordPress Alpha📲 You can test the changes from this Pull Request in WordPress Alpha by scanning the QR code below to install the corresponding build.
App NameWordPress Alpha WordPress Alpha
ConfigurationRelease-Alpha
Build Numberpr21200-7bc3089
Version22.9
Bundle IDorg.wordpress.alpha
Commit7bc3089
App Center BuildWPiOS - One-Offs #6524
Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

@jostnes jostnes marked this pull request as ready for review July 28, 2023 09:10
@jostnes jostnes requested a review from a team as a code owner July 28, 2023 09:10
@wpmobilebot
Copy link
Copy Markdown
Contributor

Jetpack Alpha📲 You can test the changes from this Pull Request in Jetpack Alpha by scanning the QR code below to install the corresponding build.
App NameJetpack Alpha Jetpack Alpha
ConfigurationRelease-Alpha
Build Numberpr21200-7bc3089
Version22.9
Bundle IDcom.jetpack.alpha
Commit7bc3089
App Center Buildjetpack-installable-builds #5555
Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

Copy link
Copy Markdown
Contributor

@pachlava pachlava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jostnes 👋

Thanks for working on this! While in theory the change should not reduce the flakiness, because the change is about making MeTabScreen be instantiated separately, instead of having it as a return value of goToMeScreen 🤔 (correct me please, if I'm missing something), let give it a go and see how it runs. The CI is green, so :shipit:

P.S. While launching the Simulator locally for the first time, this was blocking the execution:

Screenshot 2023-07-31 at 12 33 30

It looks like this does not happen on CI, though.

@jostnes
Copy link
Copy Markdown
Contributor Author

jostnes commented Jul 31, 2023

@pachlava thank you for reviewing!

While in theory the change should not reduce the flakiness, because the change is about making MeTabScreen be instantiated separately, instead of having it as a return value of goToMeScreen 🤔 (correct me please, if I'm missing something), let give it a go and see how it runs. The CI is green, so :shipit:

yes that is correct, doesn't really make sense in theory (because it should already work as is! 😅) so a bit of experimenting to see if separating it would help with the flakiness on iPad

P.S. While launching the Simulator locally for the first time, this was blocking the execution:

Screenshot 2023-07-31 at 12 33 30

It looks like this does not happen on CI, though.

that looks new, never seen it. will monitor and see if we need to handle that soon...

@jostnes jostnes merged commit f3a6cc7 into trunk Jul 31, 2023
@jostnes jostnes deleted the fix-flaky-ipad-appSettings-cell branch July 31, 2023 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Unit and UI Tests and Tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants