Example groups can override the default ordering #1025

alindeman · 2013-07-28T15:21:15Z

So ... is this even a good solution and good way to design it? 😄

There's a decent bit of weirdness around how ordering works right now
that we're trying not to break. If you see some weirdness in our code,
feel free to ask about it ... and we might decide to simply break it for
RSpec 3 to allow the solution to be cleaner.

Still outstanding if so:

OrderingRegistry has some structural duplication between
getting/setting ordering for examples vs. groups
We have not yet written rdoc/yard for the public interface for
OrderingRegistry. Some of these methods might be @private.

@samphippen and I paired on this

coveralls · 2013-07-28T15:24:04Z

Coverage increased (+0%) when pulling 7d3b260 on overriding-ordering-in-example-groups into a04fb17 on master.

myronmarston · 2013-07-31T06:39:46Z

lib/rspec/core/ordering/random_ordering.rb

+      true
+    end
+  end
+end


Given ruby 1.9's well-known perf problems with require, I think it's better to not split things into such small files. What do you think about defining each of these ordering strategies in-line in the ordering_registry.rb file?

Why don't we require the file only when it's set as the strategy, I mean we're not going to need all of them at once right?

Why don't we require the file only when it's set as the strategy, I mean we're not going to need all of them at once right?

I think that would complicate things in the registry, where it registers the strategy by class -- it would need to always load a couple of files to be able to register the built-in orderings. Seems simpler to just put them all in the same file as the registry. There's no rule that says we have to limit a file to one class :).

coveralls · 2013-07-31T12:55:36Z

Coverage increased (+0%) when pulling 2845392 on overriding-ordering-in-example-groups into a04fb17 on master.

myronmarston · 2013-07-31T15:54:11Z

spec/rspec/core/ordering/random_ordering_spec.rb

+
+    it 'resets random number generation' do
+      allow(Kernel).to receive(:srand)
+      expect(Kernel).to receive(:srand).with(no_args)


Same here...

I feel like I'm missing some context here.... (Same here as what?)

Weird, I posted a comment on the example above and github ate it or something.

My comment was: why use allow and expect? Seems like you could use just expect...

Yeah you don't need the allow on top of the expect

myronmarston · 2013-07-31T15:56:10Z

There's a decent bit of weirdness around how ordering works right now
that we're trying not to break. If you see some weirdness in our code,
feel free to ask about it ... and we might decide to simply break it for
RSpec 3 to allow the solution to be cleaner.

What weirdness are you referring to in particular? IMO, the main weirdness we had before was the use of an ordering module to extend onto arrays. You've gotten rid of that and I heartily approve :).

myronmarston · 2013-07-31T16:01:27Z

lib/rspec/core/example_group.rb

+          singleton_class.send(:define_method, :order) { args.last[:order] }
+        elsif parent && parent.order
+          singleton_class.send(:define_method, :order) { parent.order }
+        end


Rather than defining singleton methods like this, this feels like something that could be delegated to metadata (similar to described_class and file_path:

rspec-core/lib/rspec/core/example_group.rb

Line 49 in 505939b

delegate_to_metadata :described_class, :file_path

Thoughts?

Also, the singleton_class.send(:define_method, :order) { parent.order } bit seems very odd to me; won't ruby's method dispatch cause it to use the parent class's implementation already, w/o a need to explicitly define it to delegate to the parent?

Actually, perusing the code, I don't even see a place where this method gets used. Can you direct me to where this gets used? Maybe it's not even needed...

all this code should go it's a leftover from my original solution that somehow made its way into this PR.

fables-tales · 2013-07-31T16:01:47Z

The mismatch between groups and examples, also a single @order on configuration which is set depending on which of groups and examples had their ordering system set last.

myronmarston · 2013-07-31T16:02:24Z

lib/rspec/core/example_group.rb

@@ -405,9 +416,25 @@ def self.run(reporter)
        end
      end

+      # :order => MyStrategy.new
+      #
+      # :order => :my_stragegy


It's not clear what these :order => ... lines are supposed to mean, given that this method takes no args.

Also, s/stragegy/strategy/

Yeah I think this is in the wrong place

myronmarston · 2013-07-31T16:19:05Z

The mismatch between groups and examples, also a single @order on configuration which is set depending on which of groups and examples had their ordering system set last.

What is this in reply to, @samphippen? It's just floating there in the PR discussion view but I don't understand the context...

fables-tales · 2013-08-01T09:01:16Z

@myronmarston I know you're AFK but when you get back, my floating comment was in reply to:

There's a decent bit of weirdness around how ordering works right now
that we're trying not to break. If you see some weirdness in our code,
feel free to ask about it ... and we might decide to simply break it for
RSpec 3 to allow the solution to be cleaner.
What weirdness are you referring to in particular? IMO, the main weirdness we had before was the use of an ordering module to extend onto arrays. You've gotten rid of that and I heartily approve :).```

myronmarston · 2013-09-06T06:15:12Z

I'm going to try to rebase this and get this ready to merge, FWIW. I'll be force pushing the rebase shortly and then taking care of addressing the open issues above after that.

coveralls · 2013-09-07T04:39:41Z

Coverage increased (+14.39%) when pulling f4acb57 on overriding-ordering-in-example-groups into 5f0e7a0 on master.

coveralls · 2013-09-07T22:39:35Z

Coverage increased (+14.39%) when pulling ca4d4e8 on overriding-ordering-in-example-groups into 5f0e7a0 on master.

myronmarston · 2013-09-07T23:01:04Z

BTW, I have more changes coming for this. Just wanted to push what I had so far :).

myronmarston · 2013-09-09T06:22:14Z

I push some further updates. I expect the build to fail but wanted to push it to ensure it's backed up. I'll keep working on this this week.

coveralls · 2013-09-09T06:24:58Z

Coverage increased (+14.39%) when pulling 560a5ea on overriding-ordering-in-example-groups into 5f0e7a0 on master.

coveralls · 2013-09-12T03:48:19Z

Coverage increased (+14.39%) when pulling c59466b on overriding-ordering-in-example-groups into 5f0e7a0 on master.

coveralls · 2013-09-14T23:03:45Z

Coverage decreased (-0.07%) when pulling 1f887ec on overriding-ordering-in-example-groups into 3900da7 on master.

coveralls · 2013-09-16T05:33:47Z

Coverage increased (+14.42%) when pulling 727a0be on overriding-ordering-in-example-groups into 4c51980 on master.

coveralls · 2013-09-16T05:57:23Z

Coverage decreased (-0.0%) when pulling d490c75 on overriding-ordering-in-example-groups into 4c51980 on master.

coveralls · 2013-09-16T06:19:27Z

Coverage decreased (-0.04%) when pulling 9d4a982 on overriding-ordering-in-example-groups into 4c51980 on master.

coveralls · 2013-09-17T06:04:08Z

Coverage increased (+0.02%) when pulling 954d16d on overriding-ordering-in-example-groups into 4c51980 on master.

coveralls · 2013-09-18T01:58:30Z

Coverage increased (+14.42%) when pulling abdf355 on overriding-ordering-in-example-groups into 4c51980 on master.

coveralls · 2013-09-18T02:21:12Z

Coverage increased (+14.42%) when pulling 8b87d9b on overriding-ordering-in-example-groups into 4c51980 on master.

myronmarston · 2013-09-18T02:29:14Z

I've gotten this to a point that's (almost) ready to merge and I'd like feedback from others. There's still a few open issues/questions I have, though:

The config API now has order_groups/register_group_ordering, order_examples/register_example_ordering and order_groups_and_examples/register_group_and_example_ordering. For each pair, the former one is the existing API from RSpec 2 that allows users to set custom global orderings. The latter is new and allows users to register named orderings that can be used with :order => <name> metadata. I'm concerned with the size of this API and would like to simplify it if we can but I'm not sure how. Also, if we do stick with this API, it's not totally clear what the difference between all these APIs are. Is there a simpler way to do it?
What additional cukes (if any) do we want?
I've removed some methods from the config API (like #order and #randomize?). We should deprecate these in 2.99 (plus anything else we rename as discussed above).

/cc @dchelimsky @alindeman @JonRowe @samphippen @soulcutter

coveralls · 2013-09-18T02:35:34Z

Coverage increased (+14.42%) when pulling 76395cb on overriding-ordering-in-example-groups into 4c51980 on master.

JonRowe · 2013-09-22T09:58:13Z

The config API now has order_groups/register_group_ordering, order_examples/register_example_ordering and order_groups_and_examples/register_group_and_example_ordering. For each pair, the former one is the existing API from RSpec 2 that allows users to set custom global orderings. The latter is new and allows users to register named orderings that can be used with :order => <name> metadata. I'm concerned with the size of this API and would like to simplify it if we can but I'm not sure how. Also, if we do stick with this API, it's not totally clear what the difference between all these APIs are. Is there a simpler way to do it?

What is the difference, I'm not totally clued up on this. If there's no difference should we streamline them? Deprecate the old ones? Or will they be in widespread use?

What additional cukes (if any) do we want?

If we're allowing 'custom global orderings' then we should have a cuke outlining how that works (in addition to the one for using a standard ordering).

Edit Yup, we should cuke the custom block ordering, and possibly the identify one too?

JonRowe · 2013-09-22T10:06:20Z

lib/rspec/core/ordering.rb

+    module Ordering
+      # @private
+      # The default ordering (defined order).
+      class Identity


From the name of this it isn't clear to me how it's ordered.

What's a better name then? I like Identity as it's a well-known mathematical concept for a function that returns the argument it is given...which is what this does.

So it's the identity ordering, as in it leaves the thing in the same order. We were also thing about InPlace if you like that better. Thoughts on the back of a github comment ❤️

I like Identity for exactly what @myronmarston said. It's what I would have called this class.

My mathematical concepts must be rusty :P, I prefer @samphippen's InPlace suggestion.

I like Identity much better than InPlace, so let's stick with that. Honestly, it doesn't matter that much since this is an internal class that users won't need to deal with directly.

coveralls · 2013-09-27T16:29:06Z

Coverage increased (+14.48%) when pulling e7eaa57 on overriding-ordering-in-example-groups into 3091424 on master.

…rather than having special `set_global_ordering` and `global_ordering` methods. Also, simplify `Ordering::Registry#fetch`: - No need to handle `nil`; the caller can pass :global. - No need to handle callables; that's a bit of complexity we don't need. Callables can be registered. - No need to convert the ordering name `to_sym`; it wasn't spec'd and I'd rather require folks pass symbols.

- Don't have separate `groups`, `examples` and `groups_and_examples` methods. There are very few use cases for having separate logic for groups vs examples and it doesn't justify the added complexity to have separate APIs. If you need to order them separately, put a conditional in your ordering strategy that branches based on the type of the first item in the list. - Remove legacy `order_xyz` methods. These set the global ordering but didn't make that clear. Instead, the global ordering can be set by `register_ordering :global`.

`--default` was never a good name. (Consider that if we had made `--random` the default, `--default` would have been even more confusing).

coveralls · 2013-09-27T22:23:59Z

Coverage increased (+14.48%) when pulling 57ab90a on overriding-ordering-in-example-groups into 3091424 on master.

myronmarston · 2013-09-30T01:34:54Z

Circling back around on this:

First, I want to 👍--order defined. I feel it's more obvious/expressive than default.

Done, in 57ab90a.

As a side thought, what about: ...

@shepmaster I like the idea of it accepting an ordering strategy object, so I added support for that in 69c12db. However, having separate example vs group methods is similar to what we already have and are trying to get away from--plus it seems inconsistent with the block ordering form. So I decided to make everything (examples and groups) go through one API (see 10aca90). Besides the fact that this radically simplifies the API and code in rspec-core, I think that @xaviershay is right that it's very rare that these will need to be treated differently. I'm also hoping that for the cases where people actually do need to call methods on the example groups or examples, it'll put positive design pressure on us to have a unified API for them so that for the user here they can effectively be treated uniformly. Keeping them separate does not put that kind of design pressure on us.

@xaviershay I went back and forth between making the API register_ordering(:default) or register_ordering(:global) but ultimately decided on :global for a few reasons:

default is a very overloaded term. There's the use in 2.x which is now called defined. There's the use we started using it on this PR (the ordering strategy that is used by groups that don't have explicit :order metadata). But then there's the concept of what RSpec's default for that strategy is. All these uses of "default" create confusion.
I need to add the appropriate deprecation warnings to 2.99. Given that 2.x supports --order default, I think it would be confusing to have a second concept for "default" in that release.

Thus, I thought that :global was a better choice. Thoughts?

Anyhow, here's what I think is left for this feature:

Add some additional cukes (as part of this PR -- I'll try to take care of that later today or this week).
Add deprecation warnings to 2.99 for the changed APIs (as a separate PR).

xaviershay · 2013-09-30T01:36:52Z

Agree re global.

coveralls · 2013-10-01T16:35:54Z

Coverage increased (+14.48%) when pulling c5979da on overriding-ordering-in-example-groups into 3091424 on master.

myronmarston · 2013-10-01T16:37:23Z

OK, I think this is ready to merge. See my last commit for the added cukes. I opened #1092 to track the work to add the necessary deprecations to 2.99.

myronmarston · 2013-10-01T16:37:35Z

Anyone want to do a final code review before this is merged?

JonRowe · 2013-10-01T22:30:40Z

features/command_line/order.feature


  * Nested groups are always run from top-level to bottom-level in order to avoid
    executing `before(:all)` and `after(:all)` hooks more than once, but the order
    of groups at each level is randomized.

-  You can also specify a seed
+    You can also specify a seed:


Why the indentation change here?

I didn't mean to or realize I did. I'll fix that.

JonRowe · 2013-10-01T22:37:42Z

Looks mostly good to me, although I'd like to see a better solution for checking wether the seed is used rather than passing configuration into the reporter. Especially as I think we use Reporter in some of our other specs with no args.

myronmarston · 2013-10-01T22:47:50Z

Thanks for doing the review, @JonRowe.

Looks mostly good to me, although I'd like to see a better solution for checking wether the seed is used rather than passing configuration into the reporter. Especially as I think we use Reporter in some of our other specs with no args.

I agree the "seed_used?" solution could be better but I think there are more important things for me to work on. I'll merge this PR as-is once it's green (now that I addressed your other two comments) and you're welcome to open another PR with the improvements you have in mind there.

JonRowe · 2013-10-01T22:48:24Z

Yeah thats fair

coveralls · 2013-10-01T22:49:08Z

Coverage decreased (-0.03%) when pulling df880f0 on overriding-ordering-in-example-groups into 3091424 on master.

…oups Example groups can override the default ordering

myronmarston · 2013-10-01T23:33:51Z

Merged :).

myronmarston reviewed Jul 31, 2013
View reviewed changes

JonRowe reviewed Sep 22, 2013
View reviewed changes

myronmarston added 3 commits September 27, 2013 09:16

On 1.8.7, the Hash#fetch error msg doesn't include the key.

61492dc

Remove unnecessary assertion on test output.

e7eaa57

Allows ordering strategy objects to be registered.

69c12db

myronmarston added 3 commits September 27, 2013 10:21

Rename --default ordering to --defined.

57ab90a

`--default` was never a good name. (Consider that if we had made `--random` the default, `--default` would have been even more confusing).

Flesh out ordering override cuke.

c5979da

JonRowe reviewed Oct 1, 2013
View reviewed changes

myronmarston added 2 commits October 1, 2013 15:42

Fix wonky indentation.

6477ddf

Remove unneeded code.

df880f0

myronmarston added a commit that referenced this pull request Oct 1, 2013

Merge pull request #1025 from rspec/overriding-ordering-in-example-gr…

2ecfec8

…oups Example groups can override the default ordering

myronmarston merged commit 2ecfec8 into master Oct 1, 2013

myronmarston deleted the overriding-ordering-in-example-groups branch October 1, 2013 23:33

myronmarston mentioned this pull request Oct 4, 2013

Randomization is difficult to reproduce #1094

Closed

This was referenced May 31, 2016

Add additional rdoc example for ordering #2248

Merged

Rethinking ordering API and use cases #2251

Closed

Example groups can override the default ordering #1025

Example groups can override the default ordering #1025

Conversation

alindeman commented Jul 28, 2013

coveralls commented Jul 28, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jul 31, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myronmarston commented Jul 31, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fables-tales commented Jul 31, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myronmarston commented Jul 31, 2013

fables-tales commented Aug 1, 2013

myronmarston commented Sep 6, 2013

coveralls commented Sep 7, 2013

coveralls commented Sep 7, 2013

myronmarston commented Sep 7, 2013

myronmarston commented Sep 9, 2013

coveralls commented Sep 9, 2013

coveralls commented Sep 12, 2013

coveralls commented Sep 14, 2013

coveralls commented Sep 16, 2013

coveralls commented Sep 16, 2013

coveralls commented Sep 16, 2013

coveralls commented Sep 17, 2013

coveralls commented Sep 18, 2013

coveralls commented Sep 18, 2013

myronmarston commented Sep 18, 2013

coveralls commented Sep 18, 2013

JonRowe commented Sep 22, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 27, 2013

coveralls commented Sep 27, 2013

myronmarston commented Sep 30, 2013

xaviershay commented Sep 30, 2013

coveralls commented Oct 1, 2013

myronmarston commented Oct 1, 2013

myronmarston commented Oct 1, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonRowe commented Oct 1, 2013

myronmarston commented Oct 1, 2013

JonRowe commented Oct 1, 2013

coveralls commented Oct 1, 2013

myronmarston commented Oct 1, 2013