Skip to content

Increase default timeout value from 20s to 45s#16

Merged
jostnes merged 5 commits intotrunkfrom
jostnes/increase-default-timeout
Jan 9, 2025
Merged

Increase default timeout value from 20s to 45s#16
jostnes merged 5 commits intotrunkfrom
jostnes/increase-default-timeout

Conversation

@jostnes
Copy link
Copy Markdown
Contributor

@jostnes jostnes commented Jan 2, 2025

Description

There's been slowness when running tests in CI after recent CI updates. Looking at the most recent three testing builds for WCiOS, all three failed at different screens due to timeouts/element not ready/not hittable (See: 1, 2, 3)

This PR is a small change to update the default timeout value from 20 to 45 seconds. I decided on 45 seconds because some of the screens used in WCiOS use a custom timeout of 35 but still fail with timeout errors.

@jostnes jostnes changed the title Increase default timeout value from 20s to 30s Increase default timeout value from 20s to 45s Jan 8, 2025
@tiagomar
Copy link
Copy Markdown

tiagomar commented Jan 8, 2025

Checking the failures in those three links you shared and it looks like we have a few different types or errors (you actually named a few in this PR description), and I'm afraid this bump in timeout might only help with one of them.

Failed to determine hittability

Assertion Failure: Failed to determine hittability of "prologue-title-label" StaticText: Activation point invalid and no suggested hit points based on element frame

This is the one I believe the increased timeout might help with. However, we might want to get back to it and come up with a more robust and efficient solution like checking that the element dimension is not {{inf, inf}, {0.0, 0.0}} before checking it's hittability.

Observed: StaticText, 0x15631af10, {{inf, inf}, {0.0, 0.0}}, identifier: 'prologue-title-label', label: 'The ecommerce platform that grows with you’
Expected: StaticText, 0x15181a2f0, {{48.0, 449.3}, {297.0, 49.0}}, identifier: 'prologue-title-label', label: 'The ecommerce platform that grows with you'

Apple loading screen (failed: caught error: "timedOut")

This is a tricky one. I dealt with it on Day One while fixing UI Tests after iOS18. This black loading screen shows up on top of whatever app is running, shortly after the simulator is launched for the first time (on CI is always the first time). The weird part is that, even though this screen is in foreground, the application continues running and all the elements are "hittable", which allows the test runner to find the elements and try to tap them. The taps don't work but the test runner will wait for the next screen, that's when the timeouts happen.

From my experience with Day One, the loading screen usually shows up around 20-30 secs after the app is launched and last for another 30 seconds. A workaround was put in place, allowForLoadingScreenInCI(), consisting of a simple 30 seconds delay right after the app is launched for the first time before running the first test. Adding 30 seconds to a 15-20 minutes test suite didn't feel like too much.

Attempt to tap the Log In button Test fails expecting the Log In screen
Screen.Recording.2025-01-08.at.13.11.37.mov
image

Test runner crashing

WooCommerceUITests-Runner (2531) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out waiting for AX loaded notification))

We can't do much about these. ¯\_(ツ)_/¯

Copy link
Copy Markdown

@tiagomar tiagomar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @jostnes !

45 seconds sounds like a lot, but Xcode 16 and iOS 18 simulator brought some challenges. Let's keep an eye on the impacts of this change and, hopefully, revert it once Apple's issues are fixed.

@jostnes
Copy link
Copy Markdown
Contributor Author

jostnes commented Jan 9, 2025

Yup, this change might not fix all since there are a few different types but let's see if this reduces the number of flakiness at least 🤞 thanks for the review!

@jostnes jostnes merged commit 7aa2f06 into trunk Jan 9, 2025
@jostnes jostnes deleted the jostnes/increase-default-timeout branch January 9, 2025 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants