Skip to content

Allow individual example groups to override the run ordering #636

Closed
myronmarston opened this Issue Jun 14, 2012 · 17 comments

4 participants

@myronmarston
RSpec member

@rosenfeld had an interesting idea in #635: allow individual example groups to define their ordering. While I think we should keep the global --random option, being able to force one or more example groups to not run randomly is a helpful bit of flexibility, especially if you've got an existing test suite that wasn't originally written with --random in mind, and an example group or two have ordering dependencies...it lets you turn on the --random flag and then fix the order-dependent example groups over time.

In a lot of ways, this would be like minitests's i_suck_and_my_tests_are_order_dependent!, only without the hubris of the highly opinionated name.

It might be easier to do this once we've merged in #548 since it adds additional flexibility to the ordering system.

@JonRowe
RSpec member
JonRowe commented Mar 4, 2013

I'm guessing this is another, "nice to have" some day?

@myronmarston
RSpec member

Yep. I suspected that folks would find this useful as a way to migrate to the random ordering if they have projects that have ordering dependencies for some example groups, but given the lack of responses here from other users, it hasn't been a priority.

@JonRowe
RSpec member
JonRowe commented Mar 4, 2013

Understandable, I'm wondering what a sensible policy is for things like this, I mean it's obviously a feature idea and not an issue, does leaving these open harm the project's eco system? Or is this the best way to suggest and discuss such things?

@myronmarston
RSpec member

Great question. Maybe just tag them with "feature idea" or something?

@JonRowe
RSpec member
JonRowe commented Mar 4, 2013

Tagging them is definitely sensible, but I'm still wondering about how long they should be left open for, I found this via codetriage, which obviously see's all open Issues as something to be resolved, but in this case if the demand isn't there it's never going to be "resolved" as there's no point implementing features for the sake of it. Hmmz

@myronmarston
RSpec member

Any idea how other projects handle it?

@JonRowe
RSpec member
JonRowe commented Mar 4, 2013

I've seen some projects use things like public Trello boards to manage development and as such suggestions/requests for features, but that's adding another tool on top. In some situations I've seen empty pull requests used to discuss feature ideas before theres even any code, which also distances them from issues (although they still appear as issues). I guess the hard question is when to call "time" on an idea and decide not to do it?

@myronmarston
RSpec member

I kinda like using one tool for all feature/issue discussion. Does it cause any problems to keep a ticket indefinitely?

(Although, I gotta admit, getting the open issues count down to 0 would be very nice, and that is only possible if we close tickets such as these).

@JonRowe
RSpec member
JonRowe commented Mar 4, 2013

A lower number of Issue is considering healthier I guess, but maybe it doesn't cause issues.

@myronmarston
RSpec member

It'd be nice if github had explicit support for "on hold" or "idea" issues, so that it would be easy to filter the list to things you definitely plan to address while being able to keep around the ideas.

@JonRowe
RSpec member
JonRowe commented Mar 5, 2013

I'd like them to segregate pull requests more too, similarly I think your right, i'd nice to have just a 'discussion' category, keep the tagging and everything else but have no concept of open/closed, just talk with the ability to reference code, watch/unwatch etc

@rosenfeld

Just to make it clear, I'm 👍 for this. I just didn't comment here since I've already suggested this in another issue where I explained the reasons why I find this useful not only as a migration path but I find it very useful to speed up my test suite... I'm also planning on support something like this in my JS test runner, oojspec, but I haven't had much free time in the last months... :(

@myronmarston
RSpec member

Let's definitely keep it open then. I'll try to get to it in the not-too-distant future, unless someone wants to work on it and submit a pull request.... :)

I find this useful not only as a migration path but I find it very useful to speed up my test suite...

@rosenfeld, how would this speed up your test suite?

@rosenfeld

@myronmarston, let me give you a simple example. Suppose you want to test the create procedure, using the database. That consumes some time, right? Then you want to test the delete procedure, but you have to create a record in the db first, right? If I tell Rspec that the delete example depends on the create example, it will run the create example first and then the delete example that would use the record created by the prior example so that you don't have to recreate the record before deleting it. Of course, this won't work with transactional set-ups where you rollback everything after each example. But if your set up will rollback in an after-all hook for some non-transactional group you could make that group run faster if you set it up so that each example depends on the prior one being succeeded.

This is just a simple example, but I have some use cases where relying on the tests run order would speed up my suite significantly. I don't mind if all other examples will fail if the first one of the group fails, for instance. I only care to make all of them to pass.

But for my test runner oojspec it is even more complicate and it is not a matter of performance only. When you're writing integration tests for JavaScript it is common that your application will only register live events for the application life time and won't ever unregister those events. In that case, you can't simple provide some set up procedure (beforeEach, etc) to recreate some initial state. So, there should be a way of indicating which specs depend on what. So that if you want to run only spec A, it will know that it has to run B and C first.

I'm not sure if I explained it well since I'm trying to keep the explanation short. Let me know if you have any questions.

@rosenfeld

Another way to see it is like the when-xxx-and-yyy-then-aaa-and-bbb style.

@xaviershay
RSpec member

Summary: #548 has been merged, and myron has laid out a plan at the bottom of #911

@myronmarston
RSpec member

Summary: #548 has been merged, and myron has laid out a plan at the bottom of #911

Actually, #1025 is very close to being ready to merge, which implements most of #911. Closing this to focus conversation there.

BTW, @xaviershay, I'd love your feedback on the open questions in the discussion there starting here:

#1025 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.