🚀 Feature: --order option for random test order

[sarenji]

An --order option would allow people to uncover order dependencies. The three options would be --order random, --order random:seed, and --order default. Each randomized suite outputs the seed it used.

RSpec implements this, but their default order is random. Mocha doesn't have to do that. Some details on their --order parameter are here: http://blog.davidchelimsky.net/2012/01/04/rspec-28-is-released/

What do you think?

[trshafer]

+1 @yanovich. I would use a random order option which outputs a seed number. This would be very useful in a CI environment.

@visionmedia, mongoose models provide an easy example of cross-test dependencies. mongoose.model 'User', UserSchema adds a model onto the array of mongoose.models. So it's possible to create a file which relies on the user model being loaded in mongoose.models. Take Comment.find().populate('_user').exec(cb) as an example. If the user test runs before the comment test, this will execute fine, because presumably require('./models/user') (or something), has loaded the User model into mongoose.models. But if the comment test executes before the user test you'll get this error Schema hasn't been registered for model "User". This could happen in production when the comment api runs before the user api and the comment file didn't know it had a cross file dependency.

It is possible to still have the production problem with the test working if the test file has require('./models/user') (or whatever) and that loads the user into mongoose.models. However having a random order would be one more useful tool to discover potential problems like this.

I hope articulated that well. Looking forward to hearing your thoughts.

[syrnick]

+1 This is a pretty big deficiency.

rspec semantics are pretty solid: you can pass an order seed, or it can pick it at random. If it picks the seed at random, it prints it out, so it's easy to reproduce.

[boneskull]

I'll reopen this. I think it'd be helpful. While there are ways to determine cross-test deps w/o tooling, if we can automate that, it'd save people time.

After toying with this a bit, it appears non-trivial due to the hierarchical nature of Suites. Tests are run by recursing into Suites. To run Tests randomly, we'd have to enumerate them, randomize them, then work backwards.

This would cause before() and after() Hooks to be somewhat meaningless as they would get executed n times per n tests in a Suite (or rather, in the worst case, but only if we're careful), as we continually change contexts. Sounds like it'll incur a performance penalty.

Using random seeds and reporting auto-generated seeds seems trivial, however, reporters may need to know about this information, so that requires implementation(s) in the reporters.

Of course, I'm assuming what I've described here is what's being asked for. A feature like this needs a specification.

Other options include "randomize Suites" or "randomize tests within Suites" or some combination of the two. Practically speaking, this means that once you're in a describe() block A, you cannot execute tests in any parent or sibling describe() block B until all of the tests in A have been run (which looks to be a much more straightforward implementation, and won't cause hinkiness with before()/after()).

[syrnick]

What I am (and I think others are) asking for is the simplest of the options:

• randomize the tests at the lowest level: within a single describe block; shuffle "it" statements.
• randomize the order of top level suites (or randomize the order of files that get loaded)

I don't think there's much value in shuffling things at the intermediate levels.

Certainly a hack, but works for the lowest level https://github.com/syrnick/mocha/compare/random_order?expand=1&w=0

mocha - fail
connect - pass
superagent - fail
express - pass** 
websocket.io - pass (can't tell for sure)

** I got 2 intermittent failures out of 100 runs of the whole test suite either way.

[boneskull]

OK, that's certainly eaiser to implement!

I was looking at the seedrandom lib for this; use the pass option.

Would accept PR.

[boneskull]

@syrnick Mind you that if you do generate seeds, they may not be "displayable" without passing them off to reporters. I'm not very familiar with the reporting architecture, so I couldn't tell you for sure, or what to do...

[boneskull]

For those interested in this feature, they can send PRs against the randomization branch to help finish what's left.

[jhecking]

I just found the choma package, which provides a very simple plugin for Mocha to randomize the order of test suites and cases. Good alternative to rocha, which was mentioned earlier. Simple and solves the problem for me!

[Glogo]

An alternative would be to run tests in parallel:

• Karma (+ mocha or jasmine) - karma-parallel - at this moment doesn't support code coverage
• Mocha - mocha-parallel-tests or mocha.parallel
• Jest - does this by default

[JoshuaKGoldberg]

Whew, what a thread. I went back and minimized most of it as outdated. Only the relevant bits should still be left. In summary:

• After a bit of deliberation, it was decided this feature is 👍 to land in Mocha.
• The implementation should use seedrandom or an equivalent (🚀 Feature: --order option for random test order #902 (comment)).
• feat: randomization support [WIP REFERENCE - DO NOT MERGE] #2301 is part of the way to an implementation, but needs some work. See comments in the PR.

Re-marking as status: accepting prs. There's no need to explicitly comment 👍 or your use case for a random order option. If you want to push it forward, send a PR to the randomization branch. 🙂