Mark pending blocks as failed if they succeed. #1267

Merged
merged 1 commit into from Feb 10, 2014

Conversation

Projects
None yet
5 participants
@xaviershay
Member

xaviershay commented Jan 26, 2014

DO NOT MERGE: This needs final review.

See #1208 for background.

@grddev @myronmarston @JonRowe @markijbema

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Jan 26, 2014

Member

Spec failure is due to pending block not currently supporting message expectations (coming from mocks specs).

That is going to require an rspec-mocks change, let me see if that is going to be a lot of work.

Member

xaviershay commented Jan 26, 2014

Spec failure is due to pending block not currently supporting message expectations (coming from mocks specs).

That is going to require an rspec-mocks change, let me see if that is going to be a lot of work.

@xaviershay xaviershay referenced this pull request in rspec/rspec-mocks Jan 26, 2014

Closed

Wrap pending blocks in a temporary scope. #544

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Jan 26, 2014

Member

Pushed up a commit that addresses the issue, though build won't be green until rspec/rspec-mocks#544 is merged - chicken and egg problem. It would be possible with a few extra PRs to merge this in a way that keeps everything green but I don't think it's worth it.

I'm not sure whether I like this solution overall :/

Member

xaviershay commented Jan 26, 2014

Pushed up a commit that addresses the issue, though build won't be green until rspec/rspec-mocks#544 is merged - chicken and egg problem. It would be possible with a few extra PRs to merge this in a way that keeps everything green but I don't think it's worth it.

I'm not sure whether I like this solution overall :/

@markijbema

This comment has been minimized.

Show comment
Hide comment
@markijbema

markijbema Jan 26, 2014

Just to give my feedback as a user:

I actually never knew that when you used pending inside a test it had different behaviour than when you made the whole test (it) pending. Surely it's documented somewhere, but it's the sortof documentation I would look up when I wanted to know how something works, but in this case, pending already worked for me, so I never bothered to read the docs on it. So I think that improving the consistency is a clear improvement.

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

Just to give my feedback as a user:

I actually never knew that when you used pending inside a test it had different behaviour than when you made the whole test (it) pending. Surely it's documented somewhere, but it's the sortof documentation I would look up when I wanted to know how something works, but in this case, pending already worked for me, so I never bothered to read the docs on it. So I think that improving the consistency is a clear improvement.

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 27, 2014

Member

@markijbema -- thanks for engaging with us on this! It's always great to get feedback from rspec users about changes we're making.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

I've used this form of pending on many occasions. Generally, I use it when I have a spec that I've written that, due to a bug in something I'm not currently working on (e.g. a gem the project depends on, or a bit of the project that a coworker is responsible for), there's no way to get passing, but that I expect will pass in the future after I update to a newer version of a gem with a bugfix or my coworker updates their bit of the project. In situations like these, I want to keep the spec I've written, but make it pending since I want my build to stay green so that we can keep making progress. I use the block form of pending so that RSpec informs me as soon the spec starts to pass, so I know I can make it no longer pending. (This could happen during a later bundle update, for example).

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

xit isn't really intended to ever by committed (at least, not to commits you plan to keep as-is). It's intended as a quick, easy way to skip a spec: just prefix your it with an x. It's optimized for convenience over readability. I don't recommend using it for pages where you plan to push commits with some specs skipped. Before now I would have recommended using pending (as it is more readable/permanent) but now the semantics are changing a bit. Maybe we should add a skip method? That is more readable/obvious what it means/does. It would be the equivalent of ExampleGroup.pending from 2.x, and we can say that xit is a shortcut for skip.

@xaviershay -- I still owe you a full review of this. Still planning on doing that, but wanted to respond to @markijbema's feedback for now.

Member

myronmarston commented Jan 27, 2014

@markijbema -- thanks for engaging with us on this! It's always great to get feedback from rspec users about changes we're making.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

I've used this form of pending on many occasions. Generally, I use it when I have a spec that I've written that, due to a bug in something I'm not currently working on (e.g. a gem the project depends on, or a bit of the project that a coworker is responsible for), there's no way to get passing, but that I expect will pass in the future after I update to a newer version of a gem with a bugfix or my coworker updates their bit of the project. In situations like these, I want to keep the spec I've written, but make it pending since I want my build to stay green so that we can keep making progress. I use the block form of pending so that RSpec informs me as soon the spec starts to pass, so I know I can make it no longer pending. (This could happen during a later bundle update, for example).

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

xit isn't really intended to ever by committed (at least, not to commits you plan to keep as-is). It's intended as a quick, easy way to skip a spec: just prefix your it with an x. It's optimized for convenience over readability. I don't recommend using it for pages where you plan to push commits with some specs skipped. Before now I would have recommended using pending (as it is more readable/permanent) but now the semantics are changing a bit. Maybe we should add a skip method? That is more readable/obvious what it means/does. It would be the equivalent of ExampleGroup.pending from 2.x, and we can say that xit is a shortcut for skip.

@xaviershay -- I still owe you a full review of this. Still planning on doing that, but wanted to respond to @markijbema's feedback for now.

@JonRowe

This comment has been minimized.

Show comment
Hide comment
@JonRowe

JonRowe Jan 27, 2014

Member

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Writing it "will do something later" without a block will still be skipped as per usual.

Member

JonRowe commented Jan 27, 2014

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Writing it "will do something later" without a block will still be skipped as per usual.

define_method(name) do |*all_args, &block|
desc, *args = *all_args
options = Metadata.build_hash_from(args)
- options.update(:pending => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
+ options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block

This comment has been minimized.

@JonRowe

JonRowe Jan 27, 2014

Member

As we now have essentially two behaviours, "Pending" and "Skip" perhaps we should name these as such?

@JonRowe

JonRowe Jan 27, 2014

Member

As we now have essentially two behaviours, "Pending" and "Skip" perhaps we should name these as such?

lib/rspec/core/example_group.rb
@@ -105,13 +138,13 @@ def self.define_example_method(name, extra_options={})
define_example_method :pending, :pending => true
# Shortcut to define an example with :pending => 'Temporarily disabled with xexample'
# @see example
- define_example_method :xexample, :pending => 'Temporarily disabled with xexample'
+ define_example_method :xexample, :skip => 'Temporarily disabled with xexample'

This comment has been minimized.

@JonRowe

JonRowe Jan 27, 2014

Member

Why not say skipped rather than disabled.

@JonRowe

JonRowe Jan 27, 2014

Member

Why not say skipped rather than disabled.

+ end
+
+ failure_message_for_should do |example|
+ "expected: example skipped with #{message.inspect}\n got: #{example.metadata[:execution_result][:pending_message].inspect}"

This comment has been minimized.

@JonRowe

JonRowe Jan 27, 2014

Member

Perhaps we should have a skipped_message?

@JonRowe

JonRowe Jan 27, 2014

Member

Perhaps we should have a skipped_message?

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Maybe. That affects formatters, though...formatters that have known what to look for for pending would now have to be updated to look for something else. Maybe that's OK (although, if we go in that direction, it seems like we should go all the way and have both example_pending and example_skipped notifications...). However, I don't see a skipped vs pending distinction as being useful for a formatter (at least, I can't think of how it would be useful) and making that distinction adds up to a lot of work for formatter maintainers. If we're not going to make that distinction (and I'm currently leaning against doing so) then I don't think we should have a skipped_message; instead we should just treat pending vs skipped as an API/metadata concept and not as an example result/formatter concept.

What do others think about this?

/cc @alindeman @soulcutter @samphippen

@myronmarston

myronmarston Feb 1, 2014

Member

Maybe. That affects formatters, though...formatters that have known what to look for for pending would now have to be updated to look for something else. Maybe that's OK (although, if we go in that direction, it seems like we should go all the way and have both example_pending and example_skipped notifications...). However, I don't see a skipped vs pending distinction as being useful for a formatter (at least, I can't think of how it would be useful) and making that distinction adds up to a lot of work for formatter maintainers. If we're not going to make that distinction (and I'm currently leaning against doing so) then I don't think we should have a skipped_message; instead we should just treat pending vs skipped as an API/metadata concept and not as an example result/formatter concept.

What do others think about this?

/cc @alindeman @soulcutter @samphippen

This comment has been minimized.

@JonRowe

JonRowe Feb 1, 2014

Member

I've actually wanted a way to skip tests (and have them counted differently to pending) for a while, it's similar to what cucumber does....

@JonRowe

JonRowe Feb 1, 2014

Member

I've actually wanted a way to skip tests (and have them counted differently to pending) for a while, it's similar to what cucumber does....

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

I agree with @myronmarston's assessment.

@JonRowe why do you care about counting them differently? It is non-obvious to me why this is interesting.

@xaviershay

xaviershay Feb 1, 2014

Member

I agree with @myronmarston's assessment.

@JonRowe why do you care about counting them differently? It is non-obvious to me why this is interesting.

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Can you explain more what cucumber does? I don't remember. I know about @wip but that seems like just another name for our pending here; I don't remember it having an alternate "skip" idea as well.

@myronmarston

myronmarston Feb 1, 2014

Member

Can you explain more what cucumber does? I don't remember. I know about @wip but that seems like just another name for our pending here; I don't remember it having an alternate "skip" idea as well.

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Thinking about this some more, I think we can get the best of both worlds if we're careful about this. Ideally, formatters would not have to have separate handling for pending vs skipped, and could just have the current handling for pending apply to both. However, if they want to treat them differently, we should provide a mechanism to do so. I think the way to do this is to keep the single formatter notification of example_pending with the same existing metadata keys, and add a new metadata key which is pending_type which could either be :pending or :skipped -- that way, within the example_pending notification a formatter can differentiate and treat it differently if it wants, but otherwise formatters can treat them the same.

@myronmarston

myronmarston Feb 1, 2014

Member

Thinking about this some more, I think we can get the best of both worlds if we're careful about this. Ideally, formatters would not have to have separate handling for pending vs skipped, and could just have the current handling for pending apply to both. However, if they want to treat them differently, we should provide a mechanism to do so. I think the way to do this is to keep the single formatter notification of example_pending with the same existing metadata keys, and add a new metadata key which is pending_type which could either be :pending or :skipped -- that way, within the example_pending notification a formatter can differentiate and treat it differently if it wants, but otherwise formatters can treat them the same.

This comment has been minimized.

@xaviershay

xaviershay Feb 2, 2014

Member

We already have pending and skip metadata, couldn't they just use that?

@xaviershay

xaviershay Feb 2, 2014

Member

We already have pending and skip metadata, couldn't they just use that?

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Yes they could. I didn't think of that :).

@myronmarston

myronmarston Feb 4, 2014

Member

Yes they could. I didn't think of that :).

@JonRowe

This comment has been minimized.

Show comment
Hide comment
@JonRowe

JonRowe Jan 27, 2014

Member

This LGTM apart from the fact that I wonder wether we should split skip and pending more so? Also the stuff you've added to eventually allow the extension the behaviour of pending, we've not really talked about how that'd work (or even why'd you want it), and there doesn't seem to be much in the way of documentation/specs for it?

Member

JonRowe commented Jan 27, 2014

This LGTM apart from the fact that I wonder wether we should split skip and pending more so? Also the stuff you've added to eventually allow the extension the behaviour of pending, we've not really talked about how that'd work (or even why'd you want it), and there doesn't seem to be much in the way of documentation/specs for it?

@markijbema

This comment has been minimized.

Show comment
Hide comment
@markijbema

markijbema Jan 27, 2014

@myronmarston I'm only using xit for situations where I shouldn't commit it. Sometimes I don't have the time to fix it properly though, and I do. This typically involves an integration test which fails because it is written poorly, and fails even though conceptually the functionality works. Sometimes the logic is unfortunately so complex that I decide to postpone this post my pull-request, and fix it in a seperate one (for instance, one of our helpers we use in capybara tests has to be rewritten because there is a critical flaw in it; rewriting it in my own pull-request also feels wrong, because it is totally unrelated).

I like the suggestion of 'skip' though. I think that gives a nice semantic difference between skip and pending.

Also, I really like that the rspec team dares to drop backwards compatibility to improve the product. Keep up the great work :)

@myronmarston I'm only using xit for situations where I shouldn't commit it. Sometimes I don't have the time to fix it properly though, and I do. This typically involves an integration test which fails because it is written poorly, and fails even though conceptually the functionality works. Sometimes the logic is unfortunately so complex that I decide to postpone this post my pull-request, and fix it in a seperate one (for instance, one of our helpers we use in capybara tests has to be rewritten because there is a critical flaw in it; rewriting it in my own pull-request also feels wrong, because it is totally unrelated).

I like the suggestion of 'skip' though. I think that gives a nice semantic difference between skip and pending.

Also, I really like that the rspec team dares to drop backwards compatibility to improve the product. Keep up the great work :)

@@ -77,6 +77,22 @@ Feature: pending examples
And the output should contain "Expected pending 'something else getting finished' to fail. No Error was raised."
And the output should contain "pending_with_passing_block_spec.rb:3"
+ Scenario: pending any arbitrary reason, with a top-level block that passes

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I'm not sure what "a top-level block" means in this context. I tend to think of "top-level" as being in the context of main but that's not the case here. How about "Using pending to define an example that is currently passing"?

@myronmarston

myronmarston Feb 1, 2014

Member

I'm not sure what "a top-level block" means in this context. I tend to think of "top-level" as being in the context of main but that's not the case here. How about "Using pending to define an example that is currently passing"?

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

I wasn't happy with this description either.

@xaviershay

xaviershay Feb 1, 2014

Member

I wasn't happy with this description either.

lib/rspec/core/example_group.rb
+ pending
+ else
+ RSpec::Core::Pending::NO_REASON_GIVEN
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

What else would pending be besides a string? nil? If it's just a string or nil this could be simplified to:

reason = pending || RSpec::Core::Pending::NO_REASON_GIVEN

If other values (besides a string or nil) get passed, it seems odd to ignore them and odd to pass them and maybe we should do something about that...?

@myronmarston

myronmarston Feb 1, 2014

Member

What else would pending be besides a string? nil? If it's just a string or nil this could be simplified to:

reason = pending || RSpec::Core::Pending::NO_REASON_GIVEN

If other values (besides a string or nil) get passed, it seems odd to ignore them and odd to pass them and maybe we should do something about that...?

This comment has been minimized.

@JonRowe

JonRowe Feb 1, 2014

Member

👍

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

It can be a boolean: :pending => true

@xaviershay

xaviershay Feb 1, 2014

Member

It can be a boolean: :pending => true

lib/rspec/core/example_group.rb
+ RSpec::Core::Pending::NO_REASON_GIVEN
+ end
+
+ lambda {|*args|

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Given you don't use any args, it looks like the |*args| bit is just to make ruby not raise an ArgumentError since lambdas are strict about args. Instead, you can use Proc.new -- procs have the "ignore any unreceived args" semantics you're looking for.

BTW, do you follow the Weirich rule for curlies vs do/end? I've tended to adopt the more typical "do/end" for multiline blocks, curlies for single line blocks" rule that most rubyists I know use. I don't feel very strongly about it, though.

@myronmarston

myronmarston Feb 1, 2014

Member

Given you don't use any args, it looks like the |*args| bit is just to make ruby not raise an ArgumentError since lambdas are strict about args. Instead, you can use Proc.new -- procs have the "ignore any unreceived args" semantics you're looking for.

BTW, do you follow the Weirich rule for curlies vs do/end? I've tended to adopt the more typical "do/end" for multiline blocks, curlies for single line blocks" rule that most rubyists I know use. I don't feel very strongly about it, though.

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

I tend to use curlies when I care about the return value or single-line. This particular example was inconsistent of me.

@xaviershay

xaviershay Feb 1, 2014

Member

I tend to use curlies when I care about the return value or single-line. This particular example was inconsistent of me.

lib/rspec/core/example_group.rb
+ else
+ fail
+ end
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I think that this can be simplified to:

pending(reason) { instance_exec(&block) }

Assuming the purpose of fail in the else clause is to cause a failure so that the example remains pending, instance_exec(&block) will give you a failure for the block.nil? case.

@myronmarston

myronmarston Feb 1, 2014

Member

I think that this can be simplified to:

pending(reason) { instance_exec(&block) }

Assuming the purpose of fail in the else clause is to cause a failure so that the example remains pending, instance_exec(&block) will give you a failure for the block.nil? case.

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

In fact, you could probably collapse this into a one-liner:

Proc.new { pending(reason) { instance_exec(&block) } }
@myronmarston

myronmarston Feb 1, 2014

Member

In fact, you could probably collapse this into a one-liner:

Proc.new { pending(reason) { instance_exec(&block) } }
lib/rspec/core/example_group.rb
+ desc || options[:pending]
+ else
+ options[:pending]
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I think there's an open question around this...in an example like this:

pending "some string" do
  # ...
end

...should we take "some string" to be the description of the example that is being made pending, the pending reason, or both? It looks like here you are making it both. That's probably OK (the alternative is to use "no reason given" as the pending reason or whatever, and I'm not sure that's better), but if you're going to use the given string as the pending reason it should probably be set as the :pending value in the metadata as well, for consistency.

@myronmarston

myronmarston Feb 1, 2014

Member

I think there's an open question around this...in an example like this:

pending "some string" do
  # ...
end

...should we take "some string" to be the description of the example that is being made pending, the pending reason, or both? It looks like here you are making it both. That's probably OK (the alternative is to use "no reason given" as the pending reason or whatever, and I'm not sure that's better), but if you're going to use the given string as the pending reason it should probably be set as the :pending value in the metadata as well, for consistency.

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

I messed this up. "some string" is the description of the example, the default pending reason will be used, and :pending metadata will be set.

@xaviershay

xaviershay Feb 1, 2014

Member

I messed this up. "some string" is the description of the example, the default pending reason will be used, and :pending metadata will be set.

lib/rspec/core/example_group.rb
+ this.wrap_pending_block(pending, block)
+ else
+ block
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

There's a large chain of if/else in this sequence:

  • if name == :pending / else
  • if pending / else
  • within wrap_pending_block, if String === pending / else
  • again, within wrap_pending_block, if block / else

It feels unnecessarily complicated but no overarching simplifications are coming to mind, either. Any ideas if we can reduce the profusion of branching?

@myronmarston

myronmarston Feb 1, 2014

Member

There's a large chain of if/else in this sequence:

  • if name == :pending / else
  • if pending / else
  • within wrap_pending_block, if String === pending / else
  • again, within wrap_pending_block, if block / else

It feels unnecessarily complicated but no overarching simplifications are coming to mind, either. Any ideas if we can reduce the profusion of branching?

lib/rspec/core/example_group.rb
options.update(extra_options)
- examples << RSpec::Core::Example.new(self, desc, options, block)
+
+ pending = if name == :pending

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Rather than putting a "if pending was called" conditional here, can we instead move that logic into the definition of the pending method so that it doesn't have to complicate the general logic of example methods?

@myronmarston

myronmarston Feb 1, 2014

Member

Rather than putting a "if pending was called" conditional here, can we instead move that logic into the definition of the pending method so that it doesn't have to complicate the general logic of example methods?

lib/rspec/core/example_group.rb
# Shortcut to define an example with :pending => 'Temporarily disabled with xspecify'
# @see example
- define_example_method :xspecify, :pending => 'Temporarily disabled with xspecify'
+ define_example_method :xspecify, :skip => 'Temporarily disabled with xspecify'

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I think I'd also like to see an example method which is skip which would function just like the RSpec::Core::ExampleGroup.pending did in 2.x. IMO, xit, xspecify, etc are handy short cuts for when you want to temporarily skip an example (only one letter different!) but are not particularly descriptive. skip is much more descriptive about what it does, so for situations where you actually want to commit an example as skipped, the more descriptive skip form would probably be better/more intention revealing.

@myronmarston

myronmarston Feb 1, 2014

Member

I think I'd also like to see an example method which is skip which would function just like the RSpec::Core::ExampleGroup.pending did in 2.x. IMO, xit, xspecify, etc are handy short cuts for when you want to temporarily skip an example (only one letter different!) but are not particularly descriptive. skip is much more descriptive about what it does, so for situations where you actually want to commit an example as skipped, the more descriptive skip form would probably be better/more intention revealing.

lib/rspec/core/pending.rb
+ lambda { v.call(a) }
+ end
+ stack.call
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I'm trying to wrap my head around why this new pending_executors thing is needed but I'm not getting it. Can you explain? (Maybe in a committed code comment rather than as a github reply is better so we can refer to it in the future).

@myronmarston

myronmarston Feb 1, 2014

Member

I'm trying to wrap my head around why this new pending_executors thing is needed but I'm not getting it. Can you explain? (Maybe in a committed code comment rather than as a github reply is better so we can refer to it in the future).

This comment has been minimized.

@JonRowe

JonRowe Feb 1, 2014

Member

One of the other PR's @xaviershay has open utilises it to extend this.

@JonRowe

JonRowe Feb 1, 2014

Member

One of the other PR's @xaviershay has open utilises it to extend this.

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I know, I saw that. I'm trying to understand why this wasn't needed before but it is now. That's the piece I don't understand.

@xaviershay -- I'm online now and on IRC. If it's easier to explain over IRC feel free to pop in there.

@myronmarston

myronmarston Feb 1, 2014

Member

I know, I saw that. I'm trying to understand why this wasn't needed before but it is now. That's the piece I don't understand.

@xaviershay -- I'm online now and on IRC. If it's easier to explain over IRC feel free to pop in there.

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

This have never worked properly. It's required now because some specs in rspec-mocks are marked as pending so previously have never been run. Without this change they actually pass and are incorrectly failed as "fixed".

@xaviershay

xaviershay Feb 1, 2014

Member

This have never worked properly. It's required now because some specs in rspec-mocks are marked as pending so previously have never been run. Without this change they actually pass and are incorrectly failed as "fixed".

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

So it has worked in most situations for years. David initially fixed this in 644e372 by adding a call to teardown_mocks_for_rspec from within the pending logic here. There's cukes there showing it work with each of the mock framework adapters.

When I changed the rspec-mocks lifecycle to have the nested/stacked space support, the extra call to teardown_mocks_for_rspec caused a problem because it meant that the setup/teardown calls were no longer balanced. I fixed that in 6661803, which attempts to translate a failure into marking :pending_fixed as false. That works for all the examples we have testing this from within rspec-core but there's an edge case I didn't think of that has now been uncovered.

The newly uncovered edge case is here in rspec-mocks:

https://github.com/rspec/rspec-mocks/blob/7b9930ba946e288c095cf9ebd3ed27d3820edb65/spec/rspec/mocks/matchers/receive_message_chain_spec.rb#L63-L73

The problem is that there are 2 separate errors that are happening. On the and_call_original line, it is raising this error:

#<RSpec::Mocks::MockExpectationError: Double is a pure test double. `and_call_original` is only available on a partial double.>

This causes the pending block to immediately abort and set :pending_fixed to false. Later, the mock object verification happens, and a second error is raised. As a result, when we deal with this here, pending_fixed is already false, so it doesn't deal with it properly.

Your solution fixes this problem, but has two downsides to it:

  • It only works for rspec-mocks, not for other mock framework adapters. While rspec-mocks is the preferred/recommended mocking library to use with rspec-core, we want to maintain feature/behavior parity as much as possible, and so we need a solution that works with other mocking libraries.
  • There's the code that's meant to address this that I added in 6661803. I don't want us to maintain two separate ways to deal with this.

I think it should be doable to update the rescue logic in 6661803 to handle this case. As a side benefit, if you fix this directly in rspec-core w/o the extra rspec-mocks PR needed, there's no PR ordering dependency :).

Let me know if you have questions.

@myronmarston

myronmarston Feb 1, 2014

Member

So it has worked in most situations for years. David initially fixed this in 644e372 by adding a call to teardown_mocks_for_rspec from within the pending logic here. There's cukes there showing it work with each of the mock framework adapters.

When I changed the rspec-mocks lifecycle to have the nested/stacked space support, the extra call to teardown_mocks_for_rspec caused a problem because it meant that the setup/teardown calls were no longer balanced. I fixed that in 6661803, which attempts to translate a failure into marking :pending_fixed as false. That works for all the examples we have testing this from within rspec-core but there's an edge case I didn't think of that has now been uncovered.

The newly uncovered edge case is here in rspec-mocks:

https://github.com/rspec/rspec-mocks/blob/7b9930ba946e288c095cf9ebd3ed27d3820edb65/spec/rspec/mocks/matchers/receive_message_chain_spec.rb#L63-L73

The problem is that there are 2 separate errors that are happening. On the and_call_original line, it is raising this error:

#<RSpec::Mocks::MockExpectationError: Double is a pure test double. `and_call_original` is only available on a partial double.>

This causes the pending block to immediately abort and set :pending_fixed to false. Later, the mock object verification happens, and a second error is raised. As a result, when we deal with this here, pending_fixed is already false, so it doesn't deal with it properly.

Your solution fixes this problem, but has two downsides to it:

  • It only works for rspec-mocks, not for other mock framework adapters. While rspec-mocks is the preferred/recommended mocking library to use with rspec-core, we want to maintain feature/behavior parity as much as possible, and so we need a solution that works with other mocking libraries.
  • There's the code that's meant to address this that I added in 6661803. I don't want us to maintain two separate ways to deal with this.

I think it should be doable to update the rescue logic in 6661803 to handle this case. As a side benefit, if you fix this directly in rspec-core w/o the extra rspec-mocks PR needed, there's no PR ordering dependency :).

Let me know if you have questions.

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

BTW, thanks for explaining -- the bit about the failure being in the rspec-mocks specs gave me what I needed to dig in and figure out why what we had before wasn't working :).

@myronmarston

myronmarston Feb 1, 2014

Member

BTW, thanks for explaining -- the bit about the failure being in the rspec-mocks specs gave me what I needed to dig in and figure out why what we had before wasn't working :).

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 1, 2014

Member

Thanks for all the comments! Pretty sure I know how to address most of them, working on it today.

Member

xaviershay commented Feb 1, 2014

Thanks for all the comments! Pretty sure I know how to address most of them, working on it today.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 1, 2014

Member

See new commits, I think they address everything.

I just rebased this and @myronmarston's new alias_example_group_to feature is failing with these changes, so need to investigate that some more.

Member

xaviershay commented Feb 1, 2014

See new commits, I think they address everything.

I just rebased this and @myronmarston's new alias_example_group_to feature is failing with these changes, so need to investigate that some more.

lib/rspec/core/example_group.rb
+ #
+ # @see RSpec::Core::Pending#pending
+ def pending(*all_args, &block)
+ desc, *args = *all_args

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

Why not just define this as:

def pending(desc, *args, &block)
  # ...
end

...rather than pending(*all_args) and then split it on the first line?

@myronmarston

myronmarston Feb 1, 2014

Member

Why not just define this as:

def pending(desc, *args, &block)
  # ...
end

...rather than pending(*all_args) and then split it on the first line?

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

needs a default of nil, but yeah that works.

@xaviershay

xaviershay Feb 1, 2014

Member

needs a default of nil, but yeah that works.

lib/rspec/core/example_group.rb
+ callback = Proc.new { pending(reason) { instance_exec(&block) } }
+
+ examples << RSpec::Core::Example.new(self, desc, options, callback)
+ examples.last

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

There's a fair bit of duplication between this and the define_method within define_example_method. I'm concerned about future maintainability as the shared bits will need to evolve together. Here's an alternate way that I think will be more maintainable:

def self.define_example_method(name, extra_options={})
  define_method(name) do |*all_args, &block|
    desc, *args = *all_args
    options = Metadata.build_hash_from(args)
    options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
    options.update(extra_options)

    if options[:pending]
      options, block = ExampleGroup.pending_metadata_and_block_for(options, block)
    end

    examples << RSpec::Core::Example.new(self, desc, options, block)
    examples.last
  end
end

def self.pending_metadata_and_block_for(options, block)
  if String === options[:pending]
    reason = options[:pending]
  else
    options = options.merge(:pending => true)
    reason = RSpec::Core::Pending::NO_REASON_GIVEN
  end

  callback = Proc.new { pending(reason, &block) } # I thikn you can forward the block to pending w/o instance_exec
  return options, callback
end

define_example_method :pending, :pending => RSpec::Core::Pending::NO_REASON_GIVEN

Basically, this inverts it: rather than making :pending => true metadata delegate to a pending class method, it replaces the metadata and block for :pending => true and makes pending just like any other example method alias, where it just adds :pending metadata.

The previously duplicated parts are no longer duplicated here.

@myronmarston

myronmarston Feb 1, 2014

Member

There's a fair bit of duplication between this and the define_method within define_example_method. I'm concerned about future maintainability as the shared bits will need to evolve together. Here's an alternate way that I think will be more maintainable:

def self.define_example_method(name, extra_options={})
  define_method(name) do |*all_args, &block|
    desc, *args = *all_args
    options = Metadata.build_hash_from(args)
    options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
    options.update(extra_options)

    if options[:pending]
      options, block = ExampleGroup.pending_metadata_and_block_for(options, block)
    end

    examples << RSpec::Core::Example.new(self, desc, options, block)
    examples.last
  end
end

def self.pending_metadata_and_block_for(options, block)
  if String === options[:pending]
    reason = options[:pending]
  else
    options = options.merge(:pending => true)
    reason = RSpec::Core::Pending::NO_REASON_GIVEN
  end

  callback = Proc.new { pending(reason, &block) } # I thikn you can forward the block to pending w/o instance_exec
  return options, callback
end

define_example_method :pending, :pending => RSpec::Core::Pending::NO_REASON_GIVEN

Basically, this inverts it: rather than making :pending => true metadata delegate to a pending class method, it replaces the metadata and block for :pending => true and makes pending just like any other example method alias, where it just adds :pending metadata.

The previously duplicated parts are no longer duplicated here.

This comment has been minimized.

@xaviershay

xaviershay Feb 1, 2014

Member

good spot! Changed in most recent commit.

@xaviershay

xaviershay Feb 1, 2014

Member

good spot! Changed in most recent commit.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 1, 2014

Member

Build failures are the same expected RSpec mock errors.

Member

xaviershay commented Feb 1, 2014

Build failures are the same expected RSpec mock errors.

spec/rspec/core/example_group_spec.rb
@@ -888,26 +888,55 @@ def define_and_run_group(define_outer_example = false)
end
end
- %w[pending xit xspecify xexample].each do |method_name|
+ describe "::pending" do

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I've mentioned this elsewhere, but I'm not a fan of using :: for message sends. I know you're being consistent, but my preference is to change all the places we work in to use the more standard . for message sends. Mind updating it?

@myronmarston

myronmarston Feb 1, 2014

Member

I've mentioned this elsewhere, but I'm not a fan of using :: for message sends. I know you're being consistent, but my preference is to change all the places we work in to use the more standard . for message sends. Mind updating it?

spec/rspec/core/example_group_spec.rb
+ before do
+ @group = ExampleGroup.describe
+ @group.pending { fail }
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 1, 2014

Member

I'm generally not a fan of before hooks and ivars. ivars spring into existence when referenced, which means you'll get a NoMethodError for nil if the ivar is renamed or typod. (Or, in the pathological case, you could get a false positive for an expectation like expect(@some_mispelled_ivar).to be_nil, where it's only nil because you misspelled it, not because the @foo = Foo.foo line in before returned nil like you expected).

Instead I would tend to favor let for this:

let(:group) { ExampleGroup.describe { pending { fail } } }

it 'generates a pending example' do
  group.run
  expect(group.examples.first).to be_pending
end

# ...

Any reason you chose the before/ivar route?

@myronmarston

myronmarston Feb 1, 2014

Member

I'm generally not a fan of before hooks and ivars. ivars spring into existence when referenced, which means you'll get a NoMethodError for nil if the ivar is renamed or typod. (Or, in the pathological case, you could get a false positive for an expectation like expect(@some_mispelled_ivar).to be_nil, where it's only nil because you misspelled it, not because the @foo = Foo.foo line in before returned nil like you expected).

Instead I would tend to favor let for this:

let(:group) { ExampleGroup.describe { pending { fail } } }

it 'generates a pending example' do
  group.run
  expect(group.examples.first).to be_pending
end

# ...

Any reason you chose the before/ivar route?

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 2, 2014

Member

See last 4 commits. The implementation of new pending behaviour feels ... uncomfortable, possibly just because this code felt uncomfortable to start with and I haven't done any work it in before. I don't totally have all the begin/rescue blocks and different states in my head...

Member

xaviershay commented Feb 2, 2014

See last 4 commits. The implementation of new pending behaviour feels ... uncomfortable, possibly just because this code felt uncomfortable to start with and I haven't done any work it in before. I don't totally have all the begin/rescue blocks and different states in my head...

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 2, 2014

Member

Turns out changing the behaviour of pending inside an example also accidentally fixed the rspec mocks problem! I added some explicit spec coverage anyways, and removed the pending executor stuff.

I think all feedback has been addressed now. There is the open issue of whether we want to provide a different interface to formatters, since that won't be a backwards incompatible change I don't think it needs to be included here.

Member

xaviershay commented Feb 2, 2014

Turns out changing the behaviour of pending inside an example also accidentally fixed the rspec mocks problem! I added some explicit spec coverage anyways, and removed the pending executor stuff.

I think all feedback has been addressed now. There is the open issue of whether we want to provide a different interface to formatters, since that won't be a backwards incompatible change I don't think it needs to be included here.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 2, 2014

Member

DO NOT MERGE Merge this one instead: #1286

Leaving this here because the commit history is likely useful for review.

Member

xaviershay commented Feb 2, 2014

DO NOT MERGE Merge this one instead: #1286

Leaving this here because the commit history is likely useful for review.

@@ -138,7 +210,7 @@ Feature: pending examples
expect(3+4).to eq(7)
end
pending do
- expect("string".reverse).to eq("gnirts")
+ fail
end
end
"""

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

This is a big cuke, and now that we are differentiating skip vs pending I think it would be good to create separate feature files for those two cases to make the difference explicit.

Would be good if the narratives explained the difference, too.

@myronmarston

myronmarston Feb 4, 2014

Member

This is a big cuke, and now that we are differentiating skip vs pending I think it would be good to create separate feature files for those two cases to make the difference explicit.

Would be good if the narratives explained the difference, too.

lib/rspec/core/example.rb
+ :full_description,
+ :location,
+ :pending,
+ :skip

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Going multi-line on this is fine to avoid the long line, but this line-per-arg format adds to the vertical scrolling distance to move around in the file (which I dislike). What do you think about this as a compromise?

delegate_to_metadata :execution_result, :file_path, :full_description,
                     :location, :pending, :skip
@myronmarston

myronmarston Feb 4, 2014

Member

Going multi-line on this is fine to avoid the long line, but this line-per-arg format adds to the vertical scrolling distance to move around in the file (which I dislike). What do you think about this as a compromise?

delegate_to_metadata :execution_result, :file_path, :full_description,
                     :location, :pending, :skip

This comment has been minimized.

@xaviershay

xaviershay Feb 8, 2014

Member

I find the vertical easier to scan, but don't really mind. Will change to your suggestion.

@xaviershay

xaviershay Feb 8, 2014

Member

I find the vertical easier to scan, but don't really mind. Will change to your suggestion.

This comment has been minimized.

@markijbema

markijbema Feb 9, 2014

A good argument for the format by @xaviershay is that you get less incidental change, which makes your git history easier to use. If this doesn't change a lot, it doesn't matter much. Personally I tended to use the style @myronmarston suggested, but am switching to what @xaviershay uses because I noticed I do get a lot of unrelated change.

(as a sidenote, I didn't think of using \ on the first line, that's a neat trick)

@markijbema

markijbema Feb 9, 2014

A good argument for the format by @xaviershay is that you get less incidental change, which makes your git history easier to use. If this doesn't change a lot, it doesn't matter much. Personally I tended to use the style @myronmarston suggested, but am switching to what @xaviershay uses because I noticed I do get a lot of unrelated change.

(as a sidenote, I didn't think of using \ on the first line, that's a neat trick)

lib/rspec/core/example.rb
+
+ result = metadata[:execution_result]
+
+ if result[:pending_fixed] == false

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Does result lack the :pending_fixed key when pending method or metadata is involved in the example? The logic here suggests that...but I generally dislike data structure schemas that have situationally optional keys. It makes writing things that consume the metadata more complex (such as a formatter). Would be nice if the metadata schema was consistent and didn't have result keys present only some times. Maybe this could use result[:pending_declared] or something else to detect that pending is being used?

Regardless, I don't consider this a merge blocker; we can address this in a later PR if you prefer.

@myronmarston

myronmarston Feb 4, 2014

Member

Does result lack the :pending_fixed key when pending method or metadata is involved in the example? The logic here suggests that...but I generally dislike data structure schemas that have situationally optional keys. It makes writing things that consume the metadata more complex (such as a formatter). Would be nice if the metadata schema was consistent and didn't have result keys present only some times. Maybe this could use result[:pending_declared] or something else to detect that pending is being used?

Regardless, I don't consider this a merge blocker; we can address this in a later PR if you prefer.

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Actually, looking below, it looks like metadata[:execution_result][:pending_message] indicates if pending is being used. So maybe this can switch to that and :pending_fixed can always be present with a true or false value?

@myronmarston

myronmarston Feb 4, 2014

Member

Actually, looking below, it looks like metadata[:execution_result][:pending_message] indicates if pending is being used. So maybe this can switch to that and :pending_fixed can always be present with a true or false value?

This comment has been minimized.

@xaviershay

xaviershay Feb 8, 2014

Member

What should pending_fixed be if the block was not marked pending though? Doesn't really make sense. Switching conditional to check pending_message is ok though.

@xaviershay

xaviershay Feb 8, 2014

Member

What should pending_fixed be if the block was not marked pending though? Doesn't really make sense. Switching conditional to check pending_message is ok though.

This comment has been minimized.

@xaviershay

xaviershay Feb 8, 2014

Member

Oh that's not what you were arguing, you did say "when pending method is involved in example". I'll make sure it's always present in that case.

@xaviershay

xaviershay Feb 8, 2014

Member

Oh that's not what you were arguing, you did say "when pending method is involved in example". I'll make sure it's always present in that case.

lib/rspec/core/pending.rb
+ raise SkipDeclaredInExample
+ end
+
+ def set_message!(example, args)

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Given how this module is mixed-in, methods here become methods in the scope of the user's specs. They don't need this method, and if they were to define their own method named this, it would cause problems.

It looks like this acts only on the given args, so can this be turned into a class method, and then you can use Pending.set_message! to call it?

Also, since this is an easy mistake to make, it would be good to add a spec that fails if someone accidentally adds a helper method like this -- basically it could assert that the instance methods of the module contain only a known whitelist of allowed methods (e.g. pending and skip).

@myronmarston

myronmarston Feb 4, 2014

Member

Given how this module is mixed-in, methods here become methods in the scope of the user's specs. They don't need this method, and if they were to define their own method named this, it would cause problems.

It looks like this acts only on the given args, so can this be turned into a class method, and then you can use Pending.set_message! to call it?

Also, since this is an easy mistake to make, it would be good to add a spec that fails if someone accidentally adds a helper method like this -- basically it could assert that the instance methods of the module contain only a known whitelist of allowed methods (e.g. pending and skip).

lib/rspec/core/pending.rb
+
+ current_example.metadata[:skip] = true
+
+ raise SkipDeclaredInExample

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Seems odd to have a blank line between each line here. I can see there's not really any logical groupings of the lines w/in this method, but I think I'd rather see them listed as consecutive lines.

@myronmarston

myronmarston Feb 4, 2014

Member

Seems odd to have a blank line between each line here. I can see there's not really any logical groupings of the lines w/in this method, but I think I'd rather see them listed as consecutive lines.

lib/rspec/core/pending.rb
no_failure = true
current_example.metadata[:pending] = false
rescue Exception => e
current_example.execution_result[:exception] = e
+ raise

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

It's unclear to me why you both store the exception and re-raise it. If it's hard to explain, let me know--I can just clone this and play with it to see what fails when this is commented out.

@myronmarston

myronmarston Feb 4, 2014

Member

It's unclear to me why you both store the exception and re-raise it. If it's hard to explain, let me know--I can just clone this and play with it to see what fails when this is commented out.

This comment has been minimized.

@xaviershay

xaviershay Feb 8, 2014

Member

There is something weird going on here. Both these lines are required, but I'm having trouble wrapping my head around the execution paths. Very likely a better way to structure things but I think would require a shuffling around of both this code and Example.

@xaviershay

xaviershay Feb 8, 2014

Member

There is something weird going on here. Both these lines are required, but I'm having trouble wrapping my head around the execution paths. Very likely a better way to structure things but I think would require a shuffling around of both this code and Example.

This comment has been minimized.

@myronmarston

myronmarston Feb 8, 2014

Member

Don't worry too much about it; we can always refactor later :).

@myronmarston

myronmarston Feb 8, 2014

Member

Don't worry too much about it; we can always refactor later :).

+ expect(group.examples.first).to be_skipped
+ end
+ end
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

I don't see any examples that show examples being skipped using metadata rather than the skip method -- since that's part of the public API I think it needs at least a basic spec even though the code paths are being covered by other specs.

@myronmarston

myronmarston Feb 4, 2014

Member

I don't see any examples that show examples being skipped using metadata rather than the skip method -- since that's part of the public API I think it needs at least a basic spec even though the code paths are being covered by other specs.

- pending "This spec requires forking to work properly, " +
- "and JRuby does not support forking"
+ skip "This spec requires forking to work properly, " +
+ "and JRuby does not support forking"

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

This file is in rspec-support now: https://github.com/rspec/rspec-support/blob/master/lib/rspec/support/spec/in_sub_process.rb

We should remove it from here, and update the pending call there.

@myronmarston

myronmarston Feb 4, 2014

Member

This file is in rspec-support now: https://github.com/rspec/rspec-support/blob/master/lib/rspec/support/spec/in_sub_process.rb

We should remove it from here, and update the pending call there.

This comment has been minimized.

@xaviershay

xaviershay Feb 8, 2014

Member

Going to that in a separate PR, otherwise we have a chicken-and-egg problem in merging.

@xaviershay

xaviershay Feb 8, 2014

Member

Going to that in a separate PR, otherwise we have a chicken-and-egg problem in merging.

spec/rspec/core/pending_example_spec.rb
+ it 'indicates the pending block was not fixed' do
+ expect(run_example.metadata[:execution_result][:pending_fixed]).to be false
+ end
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

I expected this to require changes to RSpec::Core::Example#verify_mocks to fix but it looks like you didn't make any changes there. How did you fix this w/o changing the logic there? Also, does this imply that the logic there is now superfluous and can be removed?

@myronmarston

myronmarston Feb 4, 2014

Member

I expected this to require changes to RSpec::Core::Example#verify_mocks to fix but it looks like you didn't make any changes there. How did you fix this w/o changing the logic there? Also, does this imply that the logic there is now superfluous and can be removed?

This comment has been minimized.

@xaviershay

xaviershay Feb 4, 2014

Member

It was fixed by the same code that supports pending with no block, i.e. checking for "pending_fixed" before checking for an exception in the metadata.

@xaviershay

xaviershay Feb 4, 2014

Member

It was fixed by the same code that supports pending with no block, i.e. checking for "pending_fixed" before checking for an exception in the metadata.

+ result = example.metadata[:execution_result]
+ expect(result[:pending_fixed]).to eq(true)
+ expect(result[:status]).to eq("failed")
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

There's another case here that may or may not already work:

it 'does some stuff' do
  do some setup
  pending("this expectation needs an update in external gem X to pass") do
    expect(foo).to blah
  end

  expect(bar).to blah
end

Essentially, where pending wraps some code in the middle of the example. Or is that already covered in an existing test? I just thought of it while reading your code so this spec may not actually be needed.

@myronmarston

myronmarston Feb 4, 2014

Member

There's another case here that may or may not already work:

it 'does some stuff' do
  do some setup
  pending("this expectation needs an update in external gem X to pass") do
    expect(foo).to blah
  end

  expect(bar).to blah
end

Essentially, where pending wraps some code in the middle of the example. Or is that already covered in an existing test? I just thought of it while reading your code so this spec may not actually be needed.

This comment has been minimized.

@xaviershay

xaviershay Feb 4, 2014

Member

I'll double check there is spec coverage for this.

@xaviershay

xaviershay Feb 4, 2014

Member

I'll double check there is spec coverage for this.

+ :status => "pending",
+ :pending_message => "unimplemented"
+ )
+ ])

This comment has been minimized.

@myronmarston

myronmarston Feb 4, 2014

Member

Nice use of the new composable matchers feature :).

@myronmarston

myronmarston Feb 4, 2014

Member

Nice use of the new composable matchers feature :).

This comment has been minimized.

@xaviershay

xaviershay Feb 4, 2014

Member

tbh I cargo-culted it from elsewhere in the file ;)

@xaviershay

xaviershay Feb 4, 2014

Member

tbh I cargo-culted it from elsewhere in the file ;)

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 4, 2014

Member

Looking good, @xaviershay. Your solutions here look pretty straightforward.

Member

myronmarston commented Feb 4, 2014

Looking good, @xaviershay. Your solutions here look pretty straightforward.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 4, 2014

Member

There's an interesting setting I just found that I never knew about: show_failures_in_pending_blocks, added in a9b29ee.

It's something to bear in mind as we're wrapping this up -- I haven't given it any thought before now to make sure that our changes fit with that config setting well. There's also a cuke for it:

https://github.com/rspec/rspec-core/blob/master/features/configuration/show_failures_in_pending_blocks.feature

...which contains some statements which are no longer correct with your changes in this PR (such as "by default, code in pending examples is not exercised"). At the very least we should update that cuke for accuracy.

Member

myronmarston commented Feb 4, 2014

There's an interesting setting I just found that I never knew about: show_failures_in_pending_blocks, added in a9b29ee.

It's something to bear in mind as we're wrapping this up -- I haven't given it any thought before now to make sure that our changes fit with that config setting well. There's also a cuke for it:

https://github.com/rspec/rspec-core/blob/master/features/configuration/show_failures_in_pending_blocks.feature

...which contains some statements which are no longer correct with your changes in this PR (such as "by default, code in pending examples is not exercised"). At the very least we should update that cuke for accuracy.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

Does anyone have a dev environment already set up that lets them easily regenerated all the HTML formatter fixtures? I'm not feeling motivated to install all the required rubies... Need to run the following:

GENERATE=1 bundle exec rspec spec/rspec/core/formatters/html_formatter_spec.rb:54
Member

xaviershay commented Feb 9, 2014

Does anyone have a dev environment already set up that lets them easily regenerated all the HTML formatter fixtures? I'm not feeling motivated to install all the required rubies... Need to run the following:

GENERATE=1 bundle exec rspec spec/rspec/core/formatters/html_formatter_spec.rb:54
@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

pending.rb and example.rb need another review, I changed them pretty significantly. The logic is a bit clearer now.

Member

xaviershay commented Feb 9, 2014

pending.rb and example.rb need another review, I changed them pretty significantly. The logic is a bit clearer now.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

urgh actually hold off, still a few more outstanding issues.

Member

xaviershay commented Feb 9, 2014

urgh actually hold off, still a few more outstanding issues.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

This is still a bit of mess when interacting with mocks, but for the most part you just get specs marked as pending.

See 41ab9a7#diff-3bac19fc9abdbdd90aaec18fabf14a95R211 for my current understanding of the problem.

Member

xaviershay commented Feb 9, 2014

This is still a bit of mess when interacting with mocks, but for the most part you just get specs marked as pending.

See 41ab9a7#diff-3bac19fc9abdbdd90aaec18fabf14a95R211 for my current understanding of the problem.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

Member

xaviershay commented Feb 9, 2014

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 9, 2014

Member

Does anyone have a dev environment already set up that lets them easily regenerated all the HTML formatter fixtures? I'm not feeling motivated to install all the required rubies... Need to run the following:

I think I can do this. I'll take a stab at it later tonight (I'm only online for a couple minutes right now).

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

I'm personally a fan of this feature and it's never been a maintenance burden before. It also seems like the kind of thing that would be hard to achieve in an extension gem using public APIs. I can understand your frustration, though...I'll try to take a look at things later tonight to see where things stand.

Member

myronmarston commented Feb 9, 2014

Does anyone have a dev environment already set up that lets them easily regenerated all the HTML formatter fixtures? I'm not feeling motivated to install all the required rubies... Need to run the following:

I think I can do this. I'll take a stab at it later tonight (I'm only online for a couple minutes right now).

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

I'm personally a fan of this feature and it's never been a maintenance burden before. It also seems like the kind of thing that would be hard to achieve in an extension gem using public APIs. I can understand your frustration, though...I'll try to take a look at things later tonight to see where things stand.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

I'm mostly worried about shipping an inconsistent feature. Although it's inconsistent now...

Actually, if we remove the pending-with-a-block-in-an-example feature, I think the problem goes away. You can still use skip with :if to conditionally evaluate blocks. You still get execute and fail on pass behaviour from either normal pending keyword or pending at example group scope. Thoughts?

Member

xaviershay commented Feb 9, 2014

I'm mostly worried about shipping an inconsistent feature. Although it's inconsistent now...

Actually, if we remove the pending-with-a-block-in-an-example feature, I think the problem goes away. You can still use skip with :if to conditionally evaluate blocks. You still get execute and fail on pass behaviour from either normal pending keyword or pending at example group scope. Thoughts?

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

Semantically, saying "this part of the spec is broken now, and should work in the future, but the rest of the spec still works fine and is valuable" seems unlikely and confusing to me. Instead, marking it pending halfway through (without a block) seems a lot clearer and easy to reason about.

Even in the case of an acceptance spec, executing a pending block half-way through the spec is problematic since it puts you in an unknown state.

To put it another way, I think the high-level functionality that used to be provide by pending-with-a-block ("let me know when this passes") is useful, but the old API for it is no good. By keeping the functionality but with the new API, and removing the old API, we get a much better product.

Member

xaviershay commented Feb 9, 2014

Semantically, saying "this part of the spec is broken now, and should work in the future, but the rest of the spec still works fine and is valuable" seems unlikely and confusing to me. Instead, marking it pending halfway through (without a block) seems a lot clearer and easy to reason about.

Even in the case of an acceptance spec, executing a pending block half-way through the spec is problematic since it puts you in an unknown state.

To put it another way, I think the high-level functionality that used to be provide by pending-with-a-block ("let me know when this passes") is useful, but the old API for it is no good. By keeping the functionality but with the new API, and removing the old API, we get a much better product.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 9, 2014

Member

Ah, I misunderstood what you were talking about. I thought you were talking about the feature as a whole, and not simply the "pending block from within an example" aspect (which was the only API to the feature in 2.x). Now that I get what you're saying, I think I agree: it doesn't seem needed or useful, and the new API seems much better.

Actually, there is one case where it might still be useful: for code that is being written against both the RSpec 2.x API and RSpec 3.x API. Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use. That's probably not a big deal though: I can't think of any rspec extension gems that use this (as it's generally something in a specific end-user spec, not in an extension gem). It might affect Sequel, though, as I know it's suite is meant to work on RSpec 1.x, 2.x and 3.x (see jeremyevans/sequel@101cc3a).

I don't think that's a strong enough argument for it, though....so my vote is to axe it.

Member

myronmarston commented Feb 9, 2014

Ah, I misunderstood what you were talking about. I thought you were talking about the feature as a whole, and not simply the "pending block from within an example" aspect (which was the only API to the feature in 2.x). Now that I get what you're saying, I think I agree: it doesn't seem needed or useful, and the new API seems much better.

Actually, there is one case where it might still be useful: for code that is being written against both the RSpec 2.x API and RSpec 3.x API. Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use. That's probably not a big deal though: I can't think of any rspec extension gems that use this (as it's generally something in a specific end-user spec, not in an extension gem). It might affect Sequel, though, as I know it's suite is meant to work on RSpec 1.x, 2.x and 3.x (see jeremyevans/sequel@101cc3a).

I don't think that's a strong enough argument for it, though....so my vote is to axe it.

@markijbema

This comment has been minimized.

Show comment
Hide comment
@markijbema

markijbema Feb 9, 2014

Noticed that you have to do some effort to generate those fixtures locally. Is this something which needs to be done a lot? If so, you could consider leveraging Travis to generate them for you.

Noticed that you have to do some effort to generate those fixtures locally. Is this something which needs to be done a lot? If so, you could consider leveraging Travis to generate them for you.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

I thought you were talking about the feature as a whole

Initially I was, but I changed :)

Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use.

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible. (In 2.9 this case will deprecate and recommend a change to skip or no block.)

Member

xaviershay commented Feb 9, 2014

I thought you were talking about the feature as a whole

Initially I was, but I changed :)

Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use.

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible. (In 2.9 this case will deprecate and recommend a change to skip or no block.)

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

@markijbema This would be the second time I've had to do it. They change pretty rarely. I really just need to get my dev environment set up properly...

Member

xaviershay commented Feb 9, 2014

@markijbema This would be the second time I've had to do it. They change pretty rarely. I really just need to get my dev environment set up properly...

@grddev

This comment has been minimized.

Show comment
Hide comment
@grddev

grddev Feb 9, 2014

As I said earlier, my only strong opinion is that there should be a forward-compatible transition from RSpec 2.x, that:

  • Has the new behaviour
  • Will continue to work in 3.x
  • Does not produce any deprecation warnings in 2.99

Cutting the pending block feature seems to remove this possibility, or have I misunderstood something?

grddev commented Feb 9, 2014

As I said earlier, my only strong opinion is that there should be a forward-compatible transition from RSpec 2.x, that:

  • Has the new behaviour
  • Will continue to work in 3.x
  • Does not produce any deprecation warnings in 2.99

Cutting the pending block feature seems to remove this possibility, or have I misunderstood something?

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

@grddev you can either switch to skip with a block, or just remove the block from your pending call. 2.9 will suggest both of these options.

Member

xaviershay commented Feb 9, 2014

@grddev you can either switch to skip with a block, or just remove the block from your pending call. 2.9 will suggest both of these options.

@markijbema

This comment has been minimized.

Show comment
Hide comment
@markijbema

markijbema Feb 9, 2014

@grddev I understand what you want, and I agree that it is preferable, but there isn't really a way if we want to call this feature 'pending' in 3, right? Is there a way rspec can differentiate between a pending with block which was meant to be run, and fail if succeed, and a block which was meant not to be run? The only solution I see which would be a little closer is allow something like:

pending "foo", run: true do
end

which shows that it should be run in 2.99, and have the run param be optional in 3.

Would it be an idea to make the pending inside the block deprecated from version 3 onward? I don't think it has much more value above this syntax, right? I think that would solve the transition, since then you could transition in 2.99, and transition in 3 again. It's not perfect though. I don't see how you could solve this with one version though. For a gem I maintain we did some sequential upgrades to work around problems like this (reappropriating existing methods/names). You could also do it with 2.98 and 2.99 of course, but it feels a bit like a kludge.

@grddev I understand what you want, and I agree that it is preferable, but there isn't really a way if we want to call this feature 'pending' in 3, right? Is there a way rspec can differentiate between a pending with block which was meant to be run, and fail if succeed, and a block which was meant not to be run? The only solution I see which would be a little closer is allow something like:

pending "foo", run: true do
end

which shows that it should be run in 2.99, and have the run param be optional in 3.

Would it be an idea to make the pending inside the block deprecated from version 3 onward? I don't think it has much more value above this syntax, right? I think that would solve the transition, since then you could transition in 2.99, and transition in 3 again. It's not perfect though. I don't see how you could solve this with one version though. For a gem I maintain we did some sequential upgrades to work around problems like this (reappropriating existing methods/names). You could also do it with 2.98 and 2.99 of course, but it feels a bit like a kludge.

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

HTML fixtures need re-gen ... am hoping to be able to rebase this on top of #1306 to solve that problem.

Member

xaviershay commented Feb 9, 2014

HTML fixtures need re-gen ... am hoping to be able to rebase this on top of #1306 to solve that problem.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 9, 2014

Member

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible.

Might be worth warning and/or raising an error if a block is given to pending since it could be a common mistake from people not ware of the changes who are used to 2.x behavior, but I agree, either way, not terrible.

Member

myronmarston commented Feb 9, 2014

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible.

Might be worth warning and/or raising an error if a block is given to pending since it could be a common mistake from people not ware of the changes who are used to 2.x behavior, but I agree, either way, not terrible.

Make all uses of pending consistent.
* Execute pending examples and mark as failed if they succeed.
* Removed pending with a block.
* Introduce `skip` method and metadata for old pending behaviour.

This is a backwards-incompatible change.

Implements #1208.
@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 9, 2014

Member

I'm done making changes to this, please give it a final once over.

Might be worth warning and/or raising an error if a block is given to pending.

We're already providing a warning in 2.99, I don't think it's worth the code here. In the common case you'll get flagged anyway, since the block won't execute and the example will pass, causing a failure.

Member

xaviershay commented Feb 9, 2014

I'm done making changes to this, please give it a final once over.

Might be worth warning and/or raising an error if a block is given to pending.

We're already providing a warning in 2.99, I don't think it's worth the code here. In the common case you'll get flagged anyway, since the block won't execute and the example will pass, causing a failure.

+ an example is skipped using xexample
+ # Temporarily skipped with xexample
+ # ./temporarily_skipped_spec.rb:8
+ """

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

There's a 5th way: tagging an example with :skip => "reason why". Not a merge blocker.

@myronmarston

myronmarston Feb 10, 2014

Member

There's a 5th way: tagging an example with :skip => "reason why". Not a merge blocker.

+
+ raise Pending::PendingExampleFixedError,
+ 'Expected example to fail since it is pending, but it passed.',
+ metadata[:caller]

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

Interesting...I didn't know you could pass a different backtrace to an error like this. It's probably more user friendly to pass the metadata caller like this, but it's also potentially deceptive: if the user wants to actually see where the error was raised, (so they can look into the rspec code that raised it), the backtrace won't tell them like it usually would.

I've actually been thinking of doing away with metadata[:caller] -- as you discovered, caller is surprisingly slow, and, AFAIK, it's only needed to support line number filtering, but ruby 1.9+ provides us with a much more efficient way to support that: Proc#source_location gives us the file and line number where a block originates, and is much cheaper than using caller, so I'm planning to try to move that direction after beta2 ships, at which point it won't be available to pass here anymore. Ruby 1.8.7 would still use caller but it would just be used to set metadata[:source_location] or something similar and there would no longer be a :caller key in the metadata.

How important to you is it that metadata[:caller] is passed here?

@myronmarston

myronmarston Feb 10, 2014

Member

Interesting...I didn't know you could pass a different backtrace to an error like this. It's probably more user friendly to pass the metadata caller like this, but it's also potentially deceptive: if the user wants to actually see where the error was raised, (so they can look into the rspec code that raised it), the backtrace won't tell them like it usually would.

I've actually been thinking of doing away with metadata[:caller] -- as you discovered, caller is surprisingly slow, and, AFAIK, it's only needed to support line number filtering, but ruby 1.9+ provides us with a much more efficient way to support that: Proc#source_location gives us the file and line number where a block originates, and is much cheaper than using caller, so I'm planning to try to move that direction after beta2 ships, at which point it won't be available to pass here anymore. Ruby 1.8.7 would still use caller but it would just be used to set metadata[:source_location] or something similar and there would no longer be a :caller key in the metadata.

How important to you is it that metadata[:caller] is passed here?

This comment has been minimized.

@xaviershay

xaviershay Feb 10, 2014

Member

source_location would be fine. Without this, you have no idea where your pending block is in the error output.

@xaviershay

xaviershay Feb 10, 2014

Member

source_location would be fine. Without this, you have no idea where your pending block is in the error output.

+ metadata[:execution_result][:pending_exception] = e
+ else
+ set_exception(e)
+ end

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

There are a few other places that call set_exception and in the cases that trigger those calls, if the example is pending, should it also store the exception to metadata[:execution_result][:pending_exception] = e? Maybe the pending? check should be moved directly into set_exception? Seems like anytime set an exception we'd want this treatment, not just here, right? (Might be worth adding a spec for one or more of those cases if there is indeed a bug).

@myronmarston

myronmarston Feb 10, 2014

Member

There are a few other places that call set_exception and in the cases that trigger those calls, if the example is pending, should it also store the exception to metadata[:execution_result][:pending_exception] = e? Maybe the pending? check should be moved directly into set_exception? Seems like anytime set an exception we'd want this treatment, not just here, right? (Might be worth adding a spec for one or more of those cases if there is indeed a bug).

This comment has been minimized.

@xaviershay

xaviershay Feb 17, 2014

Member

yeah this is a bug, fix on the way.

@xaviershay

xaviershay Feb 17, 2014

Member

yeah this is a bug, fix on the way.

+ # @overload skip(message, &block)
+ #
+ # Marks an example as pending and skips execution when called without a
+ # block. When called with a block, skips just that block and does not

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

I'm trying to understand what "skips just that block" means. Take this for example:

it "does three things" do
  do_thing_1
  skip { do_thing_2 }
  do_thing_3
end

Saying it "skips just that block" suggests this is equivalent to the following:

it "does three things" do
  do_thing_1
  # do_thing_2
  do_thing_3
end

...so I'm not sure I see the point of supporting blocks at all -- IMO it makes even less semantic sense than pending with a block.

@myronmarston

myronmarston Feb 10, 2014

Member

I'm trying to understand what "skips just that block" means. Take this for example:

it "does three things" do
  do_thing_1
  skip { do_thing_2 }
  do_thing_3
end

Saying it "skips just that block" suggests this is equivalent to the following:

it "does three things" do
  do_thing_1
  # do_thing_2
  do_thing_3
end

...so I'm not sure I see the point of supporting blocks at all -- IMO it makes even less semantic sense than pending with a block.

This comment has been minimized.

@xaviershay

xaviershay Feb 10, 2014

Member

You're right, it's kind of useless. I mention this in the YARD doc, it's provided as an automatic upgrade from 2.99 (so you can just switch it in from pending), but isn't really useful for new stuff.

@xaviershay

xaviershay Feb 10, 2014

Member

You're right, it's kind of useless. I mention this in the YARD doc, it's provided as an automatic upgrade from 2.99 (so you can just switch it in from pending), but isn't really useful for new stuff.

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

I still don't get what passing a block does...is it the same as commenting out the code within the block?

What I'm confused about is that I thought that "skip" is a property of the example so I don't understand how it can even apply to a block of code within an example without applying to the whole example.

@myronmarston

myronmarston Feb 10, 2014

Member

I still don't get what passing a block does...is it the same as commenting out the code within the block?

What I'm confused about is that I thought that "skip" is a property of the example so I don't understand how it can even apply to a block of code within an example without applying to the whole example.

This comment has been minimized.

@xaviershay

xaviershay Feb 17, 2014

Member

is it the same as commenting out the code within the block

yes

What I'd like to do is deprecate it post-3.0 and then remove it. We could remove it now, just means transpec would need to convert:

it  do
  pending {
    # blah
  }
end

to

it do
  skip
  # blah
end

Hrmmm that's probably fine actually, then we could remove it now.

@xaviershay

xaviershay Feb 17, 2014

Member

is it the same as commenting out the code within the block

yes

What I'd like to do is deprecate it post-3.0 and then remove it. We could remove it now, just means transpec would need to convert:

it  do
  pending {
    # blah
  }
end

to

it do
  skip
  # blah
end

Hrmmm that's probably fine actually, then we could remove it now.

This comment has been minimized.

@xaviershay

xaviershay Feb 17, 2014

Member

oh I remember why, I wanted an automatic replacement for:

it do
  pending(:if => some_predicate) {
    # Code A
  }
  # Code B
end

Could just convert that to the closest semantic equivalent though (note in the above snipped code B would not be executed if the block was pending, which is weird).

  pending if some_predicate
  # code A
  # code B
end
@xaviershay

xaviershay Feb 17, 2014

Member

oh I remember why, I wanted an automatic replacement for:

it do
  pending(:if => some_predicate) {
    # Code A
  }
  # Code B
end

Could just convert that to the closest semantic equivalent though (note in the above snipped code B would not be executed if the block was pending, which is weird).

  pending if some_predicate
  # code A
  # code B
end
+ # fail
+ # end
+ # end
+ # end

This comment has been minimized.

@myronmarston

myronmarston Feb 10, 2014

Member

BTW, thanks for putting the effort into doc'ing things so well!

@myronmarston

myronmarston Feb 10, 2014

Member

BTW, thanks for putting the effort into doc'ing things so well!

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 10, 2014

Member

Thanks for working so hard on this, @xaviershay. I left a few more comments, but any of them that are worth addressing can be addressed in a separate PR. I'm 100% OK with this being merged as is.

Member

myronmarston commented Feb 10, 2014

Thanks for working so hard on this, @xaviershay. I left a few more comments, but any of them that are worth addressing can be addressed in a separate PR. I'm 100% OK with this being merged as is.

@markijbema

This comment has been minimized.

Show comment
Hide comment

Thanks!

@xaviershay

This comment has been minimized.

Show comment
Hide comment
@xaviershay

xaviershay Feb 10, 2014

Member

Merging. Will investigate potential bugs around the set_exception, but I'd prefer to get this in beta2 to get some real usage out of this.

Member

xaviershay commented Feb 10, 2014

Merging. Will investigate potential bugs around the set_exception, but I'd prefer to get this in beta2 to get some real usage out of this.

xaviershay added a commit that referenced this pull request Feb 10, 2014

Merge pull request #1267 from rspec/issue-1208
Mark pending blocks as failed if they succeed.

@xaviershay xaviershay merged commit facc88c into master Feb 10, 2014

1 check passed

default The Travis CI build passed
Details

@yujinakayama yujinakayama referenced this pull request in yujinakayama/transpec Feb 12, 2014

Closed

Support conversion of pending examples #38

@xaviershay xaviershay referenced this pull request Feb 17, 2014

Merged

Pending improvements #1324

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment