Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically retry failing WPT tasks (once) #24769

Closed
wants to merge 2 commits into from
Closed

Automatically retry failing WPT tasks (once) #24769

wants to merge 2 commits into from

Conversation

@SimonSapin
Copy link
Member

SimonSapin commented Nov 18, 2019

Fixes #23655

Only when ./mach filter-intermittents finds failures that are not known intermittents.

This will make PRs much more likely to land without manual retry. However it will give us a lot less opportunity to see which tests fail intermittently, file issues based on test filenames, track them, etc.

@highfive
Copy link

highfive commented Nov 18, 2019

Heads up! This PR modifies the following files:

@nox
Copy link
Member

nox commented Nov 18, 2019

LGTM, will let @jdm sign it off given that's quite a change.

@SimonSapin
Copy link
Member Author

SimonSapin commented Nov 18, 2019

@jdm What do you think? You spoke in favor in #23655, but I don’t think we’d discussed the loss of tracking.

@SimonSapin
Copy link
Member Author

SimonSapin commented Nov 18, 2019

We could imagine some more automation, such as:

  • ./mach filter-intermittent uploads its filtered summary of unexpected results somewhere
  • If and when that PR ends up landing without change, promote those previous unexpected results to a queue of possible intermittents.

However if triaging not anymore part of the expected retry process, it could grow until someone gives it attention.

@jdm
Copy link
Member

jdm commented Nov 18, 2019

Yeah, I think my biggest concern here is losing the ability to determine a meaningful window in which an intermittent failure appeared. We could build some tooling based on the disabled intermittent failure reporting that already exists that only reports the filtered summary (like you suggested), and then make it someone's responsibility to check every day or two and file new issues.

@SimonSapin
Copy link
Member Author

SimonSapin commented Nov 18, 2019

Alright. I’m not planning to build that tooling soon, so let’s close this PR and keep #23655 open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants
You can’t perform that action at this time.