-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sort tests #34
Sort tests #34
Conversation
to undo the nondeterminism introduced in rspec-3.5.0 rspec/rspec-core#2253
An alternative is to use a random but deterministic order, with But I think such randomization should only be used on a noncritical build path, similarly to what we do with |
An example of a fix of order-dependent test bug is yast/yast-network#421 |
Should we really do this or let the tests fail and fix them one by one? They are not supposed to be order-dependent. |
In other words, I would vote for the alternative proposed in @mvidner's comment. |
Now I realize that it should have been debuggable even before, because rake prints out how it calls rspec. The main problem with the randomized tests is that they tend to break at inopportune moments, and it will put the team in a situation where you are fixing a bug that a colleague made but you can't even tell them "c'mon you should've ran the tests" because they did. And the benefit is IMHO small. |
@mvidner I disagree, as order dependent tests also break when you work locally with given test. And when bootloader failed, I also rerun test with exact order written in travis and see if it can be reproduced. so also 👎 for this change. |
I still disagree and think this PR is valid, but let's close it now and argue when/if the problem appears again. |
https://bugzilla.opensuse.org/show_bug.cgi?id=990267