New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove no-shuffle from framework tests, part 1: easy cases #123751
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you sooo much!
LGTM! LGTM! LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit. So I did additional test runs overnight: 500 of them, for a total of 800, all of which passed. The largest number of tests any of these files have is 104, so if any of them were to have an ordering-dependent flake it's unlikely it would be at a rate lower than 1/104 = 0.96%. (That'd be the probability of a given test coming first, or of coming immediately after another given test.) Running on 800 seeds is enough data to reject that at p<.0004. |
Thanks for verifying that. Even if there still is an interdependency or two left in there, it's at a low enough likelihood that we can just deal with it if/when it comes up, which would also expose a seed at which it happens that would let us debug it. |
…23751) Remove no-shuffle from framework tests, part 1: easy cases
Fixes part of #85160.
I have a draft branch which removes
no-shuffle
tags from all tests in the framework.This PR collects up the "easy" fixes:
tearDown
.For all of these files, I reran the tests after my changes both with the previously failing seed mentioned in the comment, and with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.
The fixes I've left for separate PRs are those where the fix is more complex, and those where I believe changes in the
flutter_test
package are appropriate in order to make it easier either to diagnose the problem or to avoid it in the first place.Pre-launch Checklist
///
).If you need help, consider asking for advice on the #hackers-new channel on Discord.