Test order

[regexident]

I think we should come up with a more sophisticated model for test execution order.

Two goals I would want to achieve by changing the current naïve logic:

Have failing tests fail earlier

For this we would need to keep some stats about the most recent test suite execution.
Based on this we would then partition the sub-blocks of each context into two groups:

1. Tests that failed
2. Tests that succeeded

The runner would sort the sub-blocks by moving the group of failing tests to the front.

Prevent tests from depending on execution order.

Within these groups then we would want to randomize the order of sub-blocks to prevent accidental dependencies on execution order.

In principle rspec does already prevent one from depending on execution order through its sophisticated environment isolation. In the case of shared mutable state (sharing a database connection behind a mutex between contexts e.g.) however rspec cannot give such guarantees.

[regexident]

Btw, both of these stats are already collected during test execution.

As such SuiteReport, ContextReport & ExampleReport already provide:

• their respective execution time
• succeeded, failed or ignored

We would "simply" have to add serialization via Serde to persist them to ./target/rspec/… (I guess?).

[mackwic]

Yes, this is a good feature, for large refactoring it's much needed.

Ruby's Rspec use this flag: https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures

With this persistence file:

 dojo1 cat spec/examples.txt
example_id                   | status | run_time        |
---------------------------- | ------ | --------------- |
./spec/main_spec.rb[1:1:1:1] | passed | 0.00086 seconds |
./spec/main_spec.rb[1:1:2:1] | failed | 0.00011 seconds |
./spec/main_spec.rb[1:1:3:1] | passed | 0.00012 seconds |
./spec/main_spec.rb[1:1:4:1] | passed | 0.00007 seconds |
./spec/main_spec.rb[1:1:5:1] | passed | 0.00008 seconds |
./spec/main_spec.rb[1:1:6:1] | passed | 0.00012 seconds |
./spec/main_spec.rb[1:1:7:1] | failed | 0.00006 seconds |
./spec/main_spec.rb[1:1:8:1] | failed | 0.00006 seconds |

With 1:2:3 meaning: the first describe, its 2nd child, its 3rd test.

[regexident]

I'll look into it when I find some time, thanks!