[WIP] Sort the tests to have reproducible test runs? #50
Conversation
| @@ -2,6 +2,8 @@ namespace :test do | |||
| desc "Runs unit tests." | |||
| task "unit" do | |||
| files = Dir["**/test/**/*_{spec,test}.rb"] | |||
| # sort the files to have reproducible runs | |||
| files.sort! | |||
There was a problem hiding this comment.
well order depend tests are pain and I am not sure if we should workaround it. We already print order for which it failed, which is good for me. So I have still same opinion :)
|
OK, after some thinking I decided to leave the random order. In the meantime I have fixed yast/yast-network#611 which was one of the random bugs. Because we actually do not know in which exact order the files will be loaded in the real run it is possible that the same bug will appear later for real users. That randomization gives us a small chance that the bug will be found earlier in OBS/Travis/Jenkins than later by the customers. So let's keep it. |
|
BTW, this is an undead PR spawned off #34 8-] |
|
Just for the record, the random order found a problem with SCR setup in sap-installation-wizard tests (fixed in yast/sap-installation-wizard#2). When I run the tests locally with So random failures are not nice but on the other hand might help to find real problems in the tests (or in the code). |
The very same package (same SHA sum!) builds in the IBS (example log), but fails in OBS in the tests (example log).
It turned out that the difference is the order of the testing files passed to rspec. Because the order is unspecified the tests are loaded in the order in which they are stored on the disk. Which means it is basically random.
This is very tricky to debug, even in the very same enviroment but with a different filesystem the result might be completely different.
Ideally the tests should be idempotent and it should not depend on the order. But this is difficult to achieve, esp. when there are hundreds of tests. And we cannot test every possible test permutations.
So the question is: Do we prefer more reliable reproducible runs or is it better to have more or less random order which might randomly break but helps to reveal possible issues in the tests?