Load framework test files in deterministic order #37987
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`Dir.glob` doesn't guarantee the order of its results: https://ruby-doc.org/core-2.6.5/Dir.html#method-c-glob > Case sensitivity depends on your system (File::FNM_CASEFOLD is > ignored), as does the order in which the results are returned. Minitest stores a list of all test cases in the order that they were defined; it shuffles them before they're run, but doesn't sort them: https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L1048 https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L156 This means that the order in which framework tests run is platform dependent, and running a test command that failed in CI locally won't necessarily reproduce the error, even when the same seed is provided. `Rake::FileList` resolves glob patterns to a sorted list of files: https://github.com/ruby/rake/blob/v13.0.1/lib/rake/file_list.rb#L408 By using `Rake::FileList` instead of `Dir.glob`, framework tests will always run in the same order when given the same seed, and reproducing order dependent CI failures will be easier.
|
Nice, great find, @eugeneius 🙌 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dir.globdoesn't guarantee the order of its results:https://ruby-doc.org/core-2.6.5/Dir.html#method-c-glob
Minitest stores a list of all test cases in the order that they were defined; it shuffles them before they're run, but doesn't sort them:
https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L1048
https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L156
This means that the order in which framework tests run is platform dependent, and running a test command that failed in CI locally won't necessarily reproduce the error, even when the same seed is provided.
Rake::FileListresolves glob patterns to a sorted list of files:https://github.com/ruby/rake/blob/v13.0.1/lib/rake/file_list.rb#L408
By using
Rake::FileListinstead ofDir.glob, framework tests will always run in the same order when given the same seed, and reproducing order dependent CI failures will be easier.