Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS x64 browser tests should consistently pass in CI without retrying the entire testing stage #38988

Open
wknapik opened this issue Jun 12, 2024 · 1 comment
Assignees
Labels
ci-concern dev-concern OS/macOS priority/P2 A bad problem. We might uplift this to the next planned release.

Comments

@wknapik
Copy link
Contributor

wknapik commented Jun 12, 2024

Description

Browser tests have their own built-in retry mechanism, but this is not sufficient on its own. We had to rebuild often enough, that we ended up adding an automated retry in CI for the entire browser test stage, including a notification whenever it's triggered. In the last 3 months we got ~160 such notifications (Slack search: "test-browser" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI) - that's just from the public pipeline.

Each retry extends the build time and increases the size of the build queue for macOS/iOS nodes, causing everyone on the dev/qa teams to wait longer.

Those tests need to consistently pass without a retry for the entire testing stage.

PS. We just added the same retry for network-audit on macos x64, for the same reason.

@wknapik
Copy link
Contributor Author

wknapik commented Jun 17, 2024

Current count (only macos x64, only public builds, excluding PRs):

browser test full stage retry notifications: 452

Slack search:

"test-browser" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI
network audit full stage retry notifications: 143

Slack search:

"audit-network" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI
full test stage (all tests) failed even after retries: 168

Slack search:

"Failed test" "[empty]" macos-x64 in:#browser-bot from:@Jenkins CI

@mihaiplesa mihaiplesa changed the title macos x64 browser tests should consistently pass in CI without retrying the entire testing stage macOS x64 browser tests should consistently pass in CI without retrying the entire testing stage Oct 22, 2024
@rebron rebron added priority/P3 The next thing for us to work on. It'll ride the trains. priority/P2 A bad problem. We might uplift this to the next planned release. and removed priority/P3 The next thing for us to work on. It'll ride the trains. labels Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-concern dev-concern OS/macOS priority/P2 A bad problem. We might uplift this to the next planned release.
Projects
Status: In progress
Development

No branches or pull requests

3 participants