Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove no-shuffle from framework tests, part 1: easy cases #123751

Merged
merged 5 commits into from Mar 30, 2023

Conversation

gnprice
Copy link
Member

@gnprice gnprice commented Mar 30, 2023

Fixes part of #85160.

I have a draft branch which removes no-shuffle tags from all tests in the framework.

This PR collects up the "easy" fixes:

For all of these files, I reran the tests after my changes both with the previously failing seed mentioned in the comment, and with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.

The fixes I've left for separate PRs are those where the fix is more complex, and those where I believe changes in the flutter_test package are appropriate in order to make it easier either to diagnose the problem or to avoid it in the first place.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@flutter-dashboard flutter-dashboard bot added f: cupertino flutter/packages/flutter/cupertino repository framework flutter/packages/flutter repository. See also f: labels. labels Mar 30, 2023
Copy link
Member

@goderbauer goderbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you sooo much!

LGTM! LGTM! LGTM!

Copy link
Contributor

@gspencergoog gspencergoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32384589-a60f0e74-c078-11e7-9bc1-e5b5287aea9d

THANK YOU!

@gnprice
Copy link
Member Author

gnprice commented Mar 30, 2023

Thanks!

For all of these files, I reran the tests after my changes […] with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.

Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit. So I did additional test runs overnight: 500 of them, for a total of 800, all of which passed.

The largest number of tests any of these files have is 104, so if any of them were to have an ordering-dependent flake it's unlikely it would be at a rate lower than 1/104 = 0.96%. (That'd be the probability of a given test coming first, or of coming immediately after another given test.) Running on 800 seeds is enough data to reject that at p<.0004.

@gnprice gnprice added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 30, 2023
@auto-submit auto-submit bot merged commit a28aae9 into flutter:master Mar 30, 2023
72 checks passed
@gnprice gnprice deleted the pr-shuffle-part1 branch March 30, 2023 18:43
@gspencergoog
Copy link
Contributor

Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit.

Thanks for verifying that. Even if there still is an interdependency or two left in there, it's at a low enough likelihood that we can just deal with it if/when it comes up, which would also expose a seed at which it happens that would let us debug it.

exaby73 pushed a commit to NevercodeHQ/flutter that referenced this pull request Apr 17, 2023
…23751)

Remove no-shuffle from framework tests, part 1: easy cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autosubmit Merge PR when tree becomes green via auto submit App f: cupertino flutter/packages/flutter/cupertino repository framework flutter/packages/flutter repository. See also f: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants