Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Feature idea: allow examples to be run in random order (like minitest) #403

Closed
myronmarston opened this Issue · 35 comments

10 participants

@myronmarston

I prefer RSpec to minitest, but minitest does have one feature I'm jealous of: it runs the tests in a random order, and prints out the random seed (which can be used as a seed for a future test run to get the same random order). This helps you avoid having ordering dependencies between tests. Many times I've accidentally had ordering dependencies I didn't know about until the specs were run on another box (say, a linux CI server) and I got failures I wasn't getting locally. Running the specs in random order would help identify these problems early.

I'd like to bring this feature to RSpec, but there are some issues to think through and figure out first:

  1. The documentation and html formatters aren't easily compatible with a random test run. These formatters create output that mirrors the nested structure of your example groups, printing out each example as it passes or fails. This is inherently incompatible with running the examples in completely random order (since it would be very confusing to have multiple examples from the same group spread out in the output).
  2. Example groups with before(:all) or after(:all) hooks must run all their contained examples as a unit. They should not be intermingled with examples from other groups.

The second issue is addressed easily enough: we can randomize the examples within example groups that contain an all hook, and randomly intersperse these groups as a unit among the rest of the examples.

The first issue is a bit more difficult. A few thoughts:

  • For a formatter like the progress formatter or the external Fuubar formatter, this isn't an issue. Since the examples are just indicated by dots, it doesn't need to mirror the ordered structure of your example groups.
  • We could potentially delay the reporting of examples to the formatters until all examples for a group have run. This would allow complete randomization, while preserving the desired structure of the output. For a non-interactive case like a CI server, this would be fine, I think. For an interactive case, this probably wouldn't work too well since people expect to see output as the examples pass or failure.
  • We could still randomize the order of the top-level example groups.
  • Presumably the randomized ordering would be an option, and we could simply decide that the option is incompatible with the documentation and html formatters. We could spit out an error or decide to ignore one or the other setting with a warning explaining the issue.

Feedback wanted!

Myron

@dchelimsky
Owner

rspec-1 has a --loadby option which defaults to file-load-order (as rspec-2 does), and accepts mtime to load most recently updated specs first. If we re-introduce this option I'd want to use the same name (--loadby) and support mtime as well.

re: the formatters, I think it's reasonable to constrain which formatters are compatible with this option. I think the most user friendly approach would be to whitelist a set of formatters, ignore formatters that are not on that list during a run, and send a message to STDERR explaining why each was ignored. We'd probably want to support a means of whitelisting custom formatters as well.

WDYT?

@myronmarston

I've been thinking that this feature should support other ordering strategies. I'm not sure if --loadby is still the right option, though--that had to do with the order the spec files were loaded, rather than the order the examples were run, right? Continuity with the old option would be nice, but I'm not sure if it makes sense here. Instead, I'm thinking of something like --orderby. We can still support values for this option like mtime. We'd also need to support an option to specify the random seed so that if you get failures for a particular seed, you can re-run with the same seed to troubleshoot. Maybe something like this?

  • rspec spec --orderby mtime # run each file's examples as a group, and order the groups by file mtim
  • rspec spec --orderby random # order the examples completely randomly using a seed chosen by rspec (based on time or whatever)
  • rspec spec --orderby random:1234 # order randomly using seed 1234

I like your solution for the formatters--it's far easier and more straightforward than any of my suggestions.

@alindeman
Collaborator

+1 for random ordering and will help implement if needed

@kaiwren

+1 for random ordering. Also happy to help implement if needed.

@dchelimsky
Owner

How about --order instead of --orderby. It's a little less noisy and quite common in other contexts.

@Juice10

+1 for random ordering

@hgmnz

+1 random ordering, came over searching issues and pulls for this specifically. Would be great to have.

@chrisgaunt

+1 for random ordering. I just worked through a problem with ordering dependencies and came looking for this

@myronmarston

I'm still planning to get to this when I have the time...I just haven't had the time recently :(. If someone else has time to work on this, go for it...otherwise hopefully I can get to it in the not-too-distant future.

@justinko

Okay everyone I've started on this.

@Juice10

Epic!

@Confusion

Yes, this would be very useful to flesh out accidental dependencies between specs. We've also noticed that rcov doesn't always use the same ordering as rspec resulting in rspec passing, but rcov failing. The first time that happens can be quite a timesink.

@alindeman
Collaborator

Will be really interested in seeing how this is implemented!

@justinko

All done and merged into master. To use it:

--randomize
--rand

Or, you can use "seed" and it will randomize for you with the seed:

--seed 123

Point your Gemfile's to master and give it a spin!

@justinko justinko closed this
@justinko justinko was assigned
@dchelimsky
Owner

Overall looks good. I tweaked the spec a bit - let me know if you have any questions.

Couple of things:

  1. The two new options need to be accounted for in drb_args
  2. When seed is set in options, rand is automatically set. I'd like to see that happen lower down (in the configuration options). I have an idea about how to do that, and I'm happy to make the change, but you're welcome to take a crack at it if you prefer.
@dchelimsky
Owner

Also - we had talked about a --strategy or --order option that support other strategies. I'd prefer that so we don't clutter up the list of options. How about something like: --order rand, --order rand:1234 (where 1234 is the seed)?

@justinko

@dchelimsky - okay I'll change it to --order, and also add support for mtime.

@justinko

Okay I've changed it to --order rand, --order random, --order random:123.

Personally, I don't see the value of ordering the specs by mtime. Which is why I initially did --randomize. If someone wants to take that up (mtime), go right ahead.

@justinko

Oh, and if someone does want to do mtime, you can use metadata[:file_path]

@dchelimsky
Owner

mtime + fail_fast is a useful combo because it likely runs the example you're working on first.

@justinko

True....but Guard solves that pretty well.

@dchelimsky
Owner

@justinko - I'd prefer the summary message "This run was randomized by the following seed: 64349" to be up at the beginning of the run, right after the filters:

No examples matched {:focus=>true}. Running all.
Run filtered excluding {:ui=>true, :ruby=>#<Proc:./spec/spec_helper.rb:72>}
Randomized with seed 64349

WDYT?

@myronmarston

I don't have a strong preference, but I think I slightly prefer it at the end, so that if I'm running with a formatter that generates lots of output (such as the documentation formatter), I'll always have the seed output. If the seed is printed at the top, and the output is so long it scrolls off the screen, then it won't help me re-run with the same seed if I can't see what the seed was.

@dchelimsky
Owner

That's fine w/ me, but right now if feels disconnected. How about moving it before the time report?

Randomized with seed 8769.
Finished in 3.7 seconds. 
789 examples, 0 failures, 2 pending
Finished in 3.7 seconds. 
Randomized with seed 8769.
789 examples, 0 failures, 2 pending
Finished in 3.7 seconds. Randomized with seed 8769.
789 examples, 0 failures, 2 pending

WDYT?

@justinko

How about adding an extra line after it?

https://skitch.com/justinko/gd1u4/1-justinko-justins-macbook-air-code-rspec-dev-repos-rspec-core-zsh

I think we need to be conscious about being able to easily copy-n-paste the seed. To me, having the seed on its own line, with no period after it, gives the best presentation to easily drap the cursor across the number.

@dchelimsky
Owner

The thing I don't like is having it at the bottom. I want the summary (789 examples, 0 failures, 2 pending) to be the last thing unless there are links to run failed examples.

@dchelimsky
Owner

And yes, I'm fine with it on its own line with no period, but if we're looking for something to copy, why not put in the whole bit to copy:

Randomized using --order rand:1234
@justinko

Another reason why having --seed is a good idea is because I think a lot of users will set their default ordering to random:

RSpec.configure {|config| config.order = :random }

If they do that, there is no reason to specify rand on the command line. They can simply pass --seed. Given the fact that we don't have "full" randomization, I'm skeptical of anyone creating another ordering option.

That's off-topic, but to respond to the "whole bit" piece, I'm fine with that. Where would that line go? Up to you.

@Confusion

I would expect --seed to imply the order to be :random, regardless of config.order settings in RSpec.configure blocks. Otherwise you would have to change the 'order' setting in your code before you could use --seed and you can expect questions from folks that don't understand why their seeded rspec invocations execute specs in the same order as their regular invocations. However, in that case, I would say --order rand:1234 is more explicit about forcing random ordering, while achieving the same result.

@dchelimsky
Owner

I agree with @confusion. Let's leave things as they are right now and consider adding --seed as an alternative (forcing --order rand). I'm going to be adding mtime at some point, and there are other options I've got stewing.

For example, we want to be able to override the order from the CLI. Let's say you've got RSpec.configure {|c| c.order = :random} in spec_helper.rb, and you want to run with the --format doc option, you'd want to force the default ordering (alpha for directories/files, declaration order within files) so you can read it. This means we need an alpha or system or just default order option as well.

As for other order algos, a couple of really nice ones are "most likely to fail" and "most recent failures" based on historical stats. We'd have to store the stats, and I have an idea about how, but that's a separate matter.

That would bring us to:

--order rand
--order rand:1234
--seed:123 # equivalent to --order rand:123
--order mtime
--order default
--order common-failures-first # -ocff
--order recent-failures-first # -orff

So I think --order is a keeper because it opens up possibilities without adding a bunch of new options.

@dchelimsky
Owner

Changed orderby to order to align CLI and RSpec.configure: 4de9738
Renamed seed_to_report to just seed: 630f54e
Added --seed 123 (equiv of --order rand:123): a116b36

@dchelimsky dchelimsky referenced this issue from a commit
@dchelimsky dchelimsky add '--order default' option
This allows you to override --order rand in a .rspec file from the CLI.
It actually worked before, but it was completely by accident - this
makes it explicit and less likely to regress in the future.

Related to #403.
9a22066
@localshred

Tried reading through all the comments, not sure I noticed it so I'll just ask. Has this been packaged up into a beta or a release yet? I very much am interested in seeing this get through. Would be a great feature also if rspec would print the files in the order it ran them (in order to debug the exact order for reproducing failing specs). Thanks for the great work guys.

@myronmarston
Owner

@localshred -- this is in the 2.8.0 rc releases. I've been using it for a while and it works great.

Would be a great feature also if rspec would print the files in the order it ran them (in order to debug the exact order for reproducing failing specs).

The randomization isn't at the file level; it's at the example group level, and, within each group, at the example level. So I don't think this makes too much sense here.

Instead, RSpec prints out the seed that was used for the randomization, and you can pass that as an option on the next run to make it run the specs in the exact same order to debug failures.

@localshred

Ahh ok, makes sense. Thanks for the update, looking forward to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.