RSpec executes examples in the order not as they're defined

[andreypanin]

I posted this question on StackOverflow: http://stackoverflow.com/questions/26653244/why-rspec-runs-examples-in-the-order-not-as-theyre-declared

In brief, even though the documentation tells that by default, examples are executed in the order as they're defined, in fact they're not. Or that documentation looks a bit misleading.

[myronmarston]

To quote my answer on stack overflow:

The documentation is inaccurate. When an example group runs, it first runs its examples, then recursively delegates to each of its child example groups. This means that the examples of a group will always run before its nested example groups, even if some nested group were defined first.

However, with --order defined, the items within each category will be run in defined order.

This is a subtlety that we didn't bear in mind when writing the docs (or even when naming the --order defined option).

As for what we should do about it...I'm going to put some thoughts about that on the issue you opened.

We have two routes we can go here:

• We can update the documentation to clarify that examples are always run before nested groups.
• We can change the way that it works so that examples and nested groups can be interleaved and will be run in their defined order.

If we wanted to do the latter, we'd need to come up with a common "runnable" interface that examples and example groups both share, and then have each group keep track of a single list of runnables as examples and example groups are defined.

This has a downside, though. It would complicate the custom ordering API. Currently the block is given a homogenous list of either examples or example groups; if we allowed them to be interleaved it would create heterogenous lists, which could complicate the logic in the block. It could potentially break some users (particularly if they have a register_ordering block that checks the type of the first item in the list and then assumes all items in the list are of that type).

I think that I favor changing the docs to clarify how it works. Thoughts?

[kevinrutherford]

I came across this issue too. I needed to ensure that certain examples ran after everything else (because they are checking the content of logfiles to look for errors in the preceding tests). I got around it by wrapping the examples in a context, so that there was no mix of examples and contexts at any one given level.

Might be worth updating the docs to be clearer, and to recommend NOT mixing examples and contexts at the same nesting level?

[cupakromer]

I think that I favor changing the docs to clarify how it works.

Agreed. 👍

recommend NOT mixing examples and contexts at the same nesting level

I mix them all the time. Stating that it is "not recommended" is not something I would feel comfortable standing behind.

@kevinrutherford From your description of the problem you faced, I would say the way you are utilizing RSpec is not a suggested path to follow. In general, our recommendation is that specs should be able to run in isolation (i.e. I can run a single spec by itself) and thus specs should not rely on other specs being run before them.

In the few instances where I do want this, they are generally integration specs. I write these as single example "flows" which have multiple expectations per example. Otherwise, I generally have the state I want setup with before hooks to make it work.

[myronmarston]

I agree with @cupakromer -- I also mix them all the time and I do recommend it if it creates the documentation output you want.

Any takers to work up a patch updating the docs?

[JonRowe]

@cupakromer I can assure you that @kevinrutherford would not have followed that path if it wasn't necessary ;)

I think we should update the doc though

[IamNaN]

I just updated my gems and the order is no longer working the way I expected. It feels sort of broken. I like to write my tests before the code. The first test should be the first to fail. I write the code to make that work, it passes, the next test fails, and so on. You know the drill.

Unless you write your spec with a really restrictive and flat notion of contexts, the first to fail could be any test in the spec file. I'm using capybara, so I'll use features and scenarios in this example:

feature 'Comments' do
  background do
    # create something to put comments on
  end

  feature 'can be created' do
    scenario 'by the original author' do
      ...
    end

    scenario 'by moderators' do
      ...
    end

    scenario 'by other users' do
      ...
    end
  end

  scenario 'can be viewed' do
    ...
  end
end

This might not be the best example, but it does clearly shows how rspec present the developer with a chicken-and-egg problem: it jumps to the viewing scenario before any of the create scenarios. I think technically, this is correctly written spec but it can't be used for TDD. From a rubist's perspective evaluating specs top-down and then recursively when a nested context is encountered is clearly most expected. It doesn't appear that's possible currently.

I totally accept that I might be missing some key point.

[JonRowe]

As above if you put the bottom scenario in a context it will work the way you expect.

[myronmarston]

The ordering you're seeing is the ordering that RSpec has always used, AFAIK. I wasn't around in the early days so it may have changed between RSpec 1 and 2, but I'm fairly sure that the current ordering (except when you run specs with --order random) is the same as it was in RSpec 2. @IamNaN, you mention that it stopped working how you expect after upgrading RSpec...can you tell us what version you were running before that worked differently?

[IamNaN]

@myronmarston I jumped a major and then some from 2.14 to 3.1. It's entirely possible I haven't encountered this problem until now, by accident.

What @JonRowe suggests would work, but I'm afraid it would become troublesome to manage over time.

Instead, maybe I need to stop using contexts, at least like this. Divvy up the behavior into different files. In the example, there would be a file for creating and another for viewing comments. The trade-off is DRY-ness (especially with regards to backgrounds) and remembering to run the dependent specs. Clarity/predictability is a win, so it seems worth it.

[myronmarston]

@myronmarston I jumped a major and then some from 2.14 to 3.1. It's entirely possible I haven't encountered this problem until now, by accident.

I put together an example project that's getting the same ordering in 2.14 and 3.1:

https://gist.github.com/myronmarston/c0fa28496034730ad363

Can you git checkout <old SHA> on one of your projects when it was on 2.14 to see if it produces the same ordering?

Anyhow, I'm open to changing the way the defined ordering works, but I've been skeptical that it's much of a real problem for users given that I didn't think it had ever changed and this issue (opened a couple months ago, after RSpec was ~9 years old) is the first I'd heard any user feedback about the ordering. Your feedback is useful as another datapoint that an RSpec user is disliking how the ordering works. We may address this in a future release if there is more widespread community demand.

[IamNaN]

I appreciate you looking into that... Browsing the old specs in the history I can see they are a lot more simple. The projects I've been working on have become more complicated and the tests more sophisticated so I'm exercising all my tools more. The project I'm on now is probably my largest in a few years and missed this issue by happenstance until today (which coincided with a gem update, so there was some synchronicity involved here). I have to wonder how many other people are sidestepping it "just because".

All else being equal, I'm really glad to have rspec in my toolkit. It's saved me untold hours and is generally a pleasure to work with. So thanks for that.

[fgarcia]

Should I assume that examples and contexts will never be at the same nesting level?

Until now I always considered :random order a mandatory option that promotes sane isolated tests

However in a project where I must restart multiple services with asynchronous expectations for a few tests, a full tear up/down for each example is very expensive. Only in this case I found very tempting having groups and examples at the same nested level, mostly to document with detail each step of the few long-integration process

[myronmarston]

For a slow integration test I'd recommend using aggregate_failures so that you can perform the slow setup/teardown only once but make many expectations and still see all failures:

https://relishapp.com/rspec/rspec-core/v/3-4/docs/expectation-framework-integration/aggregating-failures

[mib32]

Sorry, but why when i run tests like that

rspec --format d --order defined spec/features/owner/users_spec.rb spec/libs/merchant_importers_spec.rb spec/features/authorization/sign_up_spec.rb

they run in the order 3, 1, 2 (where number is the corresponding number of argument in command line call)

[myronmarston]

Because RSpec loads the spec files in alphabetical order, not in the passed order, and "spec/libs/merchant_importers_spec.rb" < "spec/features/owner/users_spec.rb" < "spec/features/authorization/sign_up_spec.rb". See #660 for the background on this. --order defined makes the specs run in the order they are defined, and they get defined as the spec files are loaded, so the the file load order determines the run order.

If you want things to run in a specific order, we provide an API to do that:

• http://rspec.info/documentation/3.4/rspec-core/RSpec/Core/Configuration.html#register_ordering-instance_method
• https://relishapp.com/rspec/rspec-core/v/3-4/docs/configuration/overriding-global-ordering