Sort specs files to be consistent

[ggrossetie]

Currently tests are sharing a global state @env.
Sorting specs files is important to make errors reproducible by everyone.

Ref #1536

[iliabylich]

Sorting specs files is important to make errors reproducible by everyone

But it also hides possible errors. Currently when you run specs in the alphabet order you get no errors. But when you run it in the reverse order you get an error in rescue_spec.

I thought it's a good practice to run specs in the random order.

MSpec has on option called randomize that shuffles files, I think it makes sense to enable it to catch possible conflicts between specs.

[ggrossetie]

You are right but I don't like random failures because it's pretty hard to reproduce and make the build unstable.
I would rather prefer a consistent behavior for every developers.

Catching bugs between specs is nice but specs are mainly designed to catch bugs in the "production" code not in the "testing" code 🐛

[iliabylich]

👍

• If we want to run a test suite in a normal (i.e. sequential) order, we simply run it as is. The order can't depend on the platform or locale(?), this is wrong.
• If we want to a random order, we add MSpec.randomize(true) to the spec_helper by setting some environment variable.

@elia Any thoughts?

[iliabylich]

By the way, there's no need to sort spec filters, they can't conflict

[elia]

Ideally we should add a shuffle after seeding srand with a number we also print out, that should make the random order reproducible, but still random.

> srand 123; (1..10).to_a.shuffle
=> [5, 1, 8, 6, 9, 4, 2, 7, 10, 3]
> srand 123; (1..10).to_a.shuffle
=> [5, 1, 8, 6, 9, 4, 2, 7, 10, 3]
> srand 123; (1..10).to_a.shuffle
=> [5, 1, 8, 6, 9, 4, 2, 7, 10, 3]

[ggrossetie]

Ideally we should add a shuffle after seeding srand with a number we also print out, that should make the random order reproducible, but still random.

Yes that's a good idea! Reproducible is the key 😄

[elia]

Yes that's a good idea! Reproducible is the key 😄

To be fair, it's RSpec team's idea💡 😄

[iliabylich]

@elia It can be solved after #1540. Can we merge this PR for now?

[elia]

sure thing