Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include wait time in more cases when UI testing progress bubble color #41676

Merged
merged 1 commit into from
Jul 22, 2021

Conversation

Erin007
Copy link
Contributor

@Erin007 Erin007 commented Jul 21, 2021

Related to X-TEAM 366

While investigating UI test failures on a Drone run I stumbled on what appears to be some flakiness in the tests covering multi page assessments. In the Drone run and then occasionally locally on a fresh pull of staging there was an error where we expected a bubble to be purple (perfect), but it was showing as white (not tried).

Screen Shot 2021-07-21 at 5 09 38 PM

There was a time-saving optimization in place to only add additional wait time if the status of the bubble was not_tried, but I think the additional wait time is needed in other cases as well to reduce flakiness in this test. These tests consistently pass for me locally when the additional wait time and completion of Ajax requests are added in as steps.

@Erin007
Copy link
Contributor Author

Erin007 commented Jul 21, 2021

I don't think any of the LP tests related to bubble color have hit our flakiness threshold, but it does look like there have been re-runs needed on a few with a similar error in the past couple of days.
Screen Shot 2021-07-21 at 5 16 51 PM

@jamescodeorg
Copy link
Contributor

Thanks for digging into this! I'm fine with switching to this algorithm, but I'm not sure I understand why this change helps.

The original code was supposed to wait up to 30s for the bubble to turn colors. From the logs, it doesn't look like this polling happened (i.e. the test gave up after the first immediate failure)?

@Erin007
Copy link
Contributor Author

Erin007 commented Jul 22, 2021

Thanks for digging into this! I'm fine with switching to this algorithm, but I'm not sure I understand why this change helps.

The original code was supposed to wait up to 30s for the bubble to turn colors. From the logs, it doesn't look like this polling happened (i.e. the test gave up after the first immediate failure)?

Hmmm. I'm not 100% sure either. There are a few other tests that use the wait_short_until step. I'll take a peek to see if it's working in those scenarios.

@Erin007 Erin007 merged commit 3e52521 into staging Jul 22, 2021
@Erin007 Erin007 deleted the multi-dot-ui-test branch July 22, 2021 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants