Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark pending blocks as failed if they succeed. #1267

Merged
merged 1 commit into from Feb 10, 2014
Merged

Mark pending blocks as failed if they succeed. #1267

merged 1 commit into from Feb 10, 2014

Conversation

@xaviershay
Copy link
Member

xaviershay commented Jan 26, 2014

DO NOT MERGE: This needs final review.

See #1208 for background.

@grddev @myronmarston @JonRowe @markijbema

@xaviershay
Copy link
Member Author

xaviershay commented Jan 26, 2014

Spec failure is due to pending block not currently supporting message expectations (coming from mocks specs).

That is going to require an rspec-mocks change, let me see if that is going to be a lot of work.

@xaviershay
Copy link
Member Author

xaviershay commented Jan 26, 2014

Pushed up a commit that addresses the issue, though build won't be green until rspec/rspec-mocks#544 is merged - chicken and egg problem. It would be possible with a few extra PRs to merge this in a way that keeps everything green but I don't think it's worth it.

I'm not sure whether I like this solution overall :/

@markijbema
Copy link

markijbema commented Jan 26, 2014

Just to give my feedback as a user:

I actually never knew that when you used pending inside a test it had different behaviour than when you made the whole test (it) pending. Surely it's documented somewhere, but it's the sortof documentation I would look up when I wanted to know how something works, but in this case, pending already worked for me, so I never bothered to read the docs on it. So I think that improving the consistency is a clear improvement.

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

@myronmarston
Copy link
Member

myronmarston commented Jan 27, 2014

@markijbema -- thanks for engaging with us on this! It's always great to get feedback from rspec users about changes we're making.

Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red?

I've used this form of pending on many occasions. Generally, I use it when I have a spec that I've written that, due to a bug in something I'm not currently working on (e.g. a gem the project depends on, or a bit of the project that a coworker is responsible for), there's no way to get passing, but that I expect will pass in the future after I update to a newer version of a gem with a bugfix or my coworker updates their bit of the project. In situations like these, I want to keep the spec I've written, but make it pending since I want my build to stay green so that we can keep making progress. I use the block form of pending so that RSpec informs me as soon the spec starts to pass, so I know I can make it no longer pending. (This could happen during a later bundle update, for example).

Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay.

xit isn't really intended to ever by committed (at least, not to commits you plan to keep as-is). It's intended as a quick, easy way to skip a spec: just prefix your it with an x. It's optimized for convenience over readability. I don't recommend using it for pages where you plan to push commits with some specs skipped. Before now I would have recommended using pending (as it is more readable/permanent) but now the semantics are changing a bit. Maybe we should add a skip method? That is more readable/obvious what it means/does. It would be the equivalent of ExampleGroup.pending from 2.x, and we can say that xit is a shortcut for skip.

@xaviershay -- I still owe you a full review of this. Still planning on doing that, but wanted to respond to @markijbema's feedback for now.

@JonRowe
Copy link
Member

JonRowe commented Jan 27, 2014

I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'.

Writing it "will do something later" without a block will still be skipped as per usual.

define_method(name) do |*all_args, &block|
desc, *args = *all_args
options = Metadata.build_hash_from(args)
options.update(:pending => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block

This comment has been minimized.

Copy link
@JonRowe

JonRowe Jan 27, 2014

Member

As we now have essentially two behaviours, "Pending" and "Skip" perhaps we should name these as such?

@@ -105,13 +138,13 @@ def self.define_example_method(name, extra_options={})
define_example_method :pending, :pending => true
# Shortcut to define an example with :pending => 'Temporarily disabled with xexample'
# @see example
define_example_method :xexample, :pending => 'Temporarily disabled with xexample'
define_example_method :xexample, :skip => 'Temporarily disabled with xexample'

This comment has been minimized.

Copy link
@JonRowe

JonRowe Jan 27, 2014

Member

Why not say skipped rather than disabled.

end

failure_message_for_should do |example|
"expected: example skipped with #{message.inspect}\n got: #{example.metadata[:execution_result][:pending_message].inspect}"

This comment has been minimized.

Copy link
@JonRowe

JonRowe Jan 27, 2014

Member

Perhaps we should have a skipped_message?

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Maybe. That affects formatters, though...formatters that have known what to look for for pending would now have to be updated to look for something else. Maybe that's OK (although, if we go in that direction, it seems like we should go all the way and have both example_pending and example_skipped notifications...). However, I don't see a skipped vs pending distinction as being useful for a formatter (at least, I can't think of how it would be useful) and making that distinction adds up to a lot of work for formatter maintainers. If we're not going to make that distinction (and I'm currently leaning against doing so) then I don't think we should have a skipped_message; instead we should just treat pending vs skipped as an API/metadata concept and not as an example result/formatter concept.

What do others think about this?

/cc @alindeman @soulcutter @samphippen

This comment has been minimized.

Copy link
@JonRowe

JonRowe Feb 1, 2014

Member

I've actually wanted a way to skip tests (and have them counted differently to pending) for a while, it's similar to what cucumber does....

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

I agree with @myronmarston's assessment.

@JonRowe why do you care about counting them differently? It is non-obvious to me why this is interesting.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Can you explain more what cucumber does? I don't remember. I know about @wip but that seems like just another name for our pending here; I don't remember it having an alternate "skip" idea as well.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Thinking about this some more, I think we can get the best of both worlds if we're careful about this. Ideally, formatters would not have to have separate handling for pending vs skipped, and could just have the current handling for pending apply to both. However, if they want to treat them differently, we should provide a mechanism to do so. I think the way to do this is to keep the single formatter notification of example_pending with the same existing metadata keys, and add a new metadata key which is pending_type which could either be :pending or :skipped -- that way, within the example_pending notification a formatter can differentiate and treat it differently if it wants, but otherwise formatters can treat them the same.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 2, 2014

Author Member

We already have pending and skip metadata, couldn't they just use that?

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 4, 2014

Member

Yes they could. I didn't think of that :).

@JonRowe
Copy link
Member

JonRowe commented Jan 27, 2014

This LGTM apart from the fact that I wonder wether we should split skip and pending more so? Also the stuff you've added to eventually allow the extension the behaviour of pending, we've not really talked about how that'd work (or even why'd you want it), and there doesn't seem to be much in the way of documentation/specs for it?

@markijbema
Copy link

markijbema commented Jan 27, 2014

@myronmarston I'm only using xit for situations where I shouldn't commit it. Sometimes I don't have the time to fix it properly though, and I do. This typically involves an integration test which fails because it is written poorly, and fails even though conceptually the functionality works. Sometimes the logic is unfortunately so complex that I decide to postpone this post my pull-request, and fix it in a seperate one (for instance, one of our helpers we use in capybara tests has to be rewritten because there is a critical flaw in it; rewriting it in my own pull-request also feels wrong, because it is totally unrelated).

I like the suggestion of 'skip' though. I think that gives a nice semantic difference between skip and pending.

Also, I really like that the rspec team dares to drop backwards compatibility to improve the product. Keep up the great work :)

@@ -77,6 +77,22 @@ Feature: pending examples
And the output should contain "Expected pending 'something else getting finished' to fail. No Error was raised."
And the output should contain "pending_with_passing_block_spec.rb:3"

Scenario: pending any arbitrary reason, with a top-level block that passes

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I'm not sure what "a top-level block" means in this context. I tend to think of "top-level" as being in the context of main but that's not the case here. How about "Using pending to define an example that is currently passing"?

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

I wasn't happy with this description either.

pending
else
RSpec::Core::Pending::NO_REASON_GIVEN
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

What else would pending be besides a string? nil? If it's just a string or nil this could be simplified to:

reason = pending || RSpec::Core::Pending::NO_REASON_GIVEN

If other values (besides a string or nil) get passed, it seems odd to ignore them and odd to pass them and maybe we should do something about that...?

This comment has been minimized.

Copy link
@JonRowe

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

It can be a boolean: :pending => true

RSpec::Core::Pending::NO_REASON_GIVEN
end

lambda {|*args|

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Given you don't use any args, it looks like the |*args| bit is just to make ruby not raise an ArgumentError since lambdas are strict about args. Instead, you can use Proc.new -- procs have the "ignore any unreceived args" semantics you're looking for.

BTW, do you follow the Weirich rule for curlies vs do/end? I've tended to adopt the more typical "do/end" for multiline blocks, curlies for single line blocks" rule that most rubyists I know use. I don't feel very strongly about it, though.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

I tend to use curlies when I care about the return value or single-line. This particular example was inconsistent of me.

else
fail
end
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I think that this can be simplified to:

pending(reason) { instance_exec(&block) }

Assuming the purpose of fail in the else clause is to cause a failure so that the example remains pending, instance_exec(&block) will give you a failure for the block.nil? case.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

In fact, you could probably collapse this into a one-liner:

Proc.new { pending(reason) { instance_exec(&block) } }
desc || options[:pending]
else
options[:pending]
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I think there's an open question around this...in an example like this:

pending "some string" do
  # ...
end

...should we take "some string" to be the description of the example that is being made pending, the pending reason, or both? It looks like here you are making it both. That's probably OK (the alternative is to use "no reason given" as the pending reason or whatever, and I'm not sure that's better), but if you're going to use the given string as the pending reason it should probably be set as the :pending value in the metadata as well, for consistency.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

I messed this up. "some string" is the description of the example, the default pending reason will be used, and :pending metadata will be set.

this.wrap_pending_block(pending, block)
else
block
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

There's a large chain of if/else in this sequence:

  • if name == :pending / else
  • if pending / else
  • within wrap_pending_block, if String === pending / else
  • again, within wrap_pending_block, if block / else

It feels unnecessarily complicated but no overarching simplifications are coming to mind, either. Any ideas if we can reduce the profusion of branching?

options.update(extra_options)
examples << RSpec::Core::Example.new(self, desc, options, block)

pending = if name == :pending

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Rather than putting a "if pending was called" conditional here, can we instead move that logic into the definition of the pending method so that it doesn't have to complicate the general logic of example methods?

# Shortcut to define an example with :pending => 'Temporarily disabled with xspecify'
# @see example
define_example_method :xspecify, :pending => 'Temporarily disabled with xspecify'
define_example_method :xspecify, :skip => 'Temporarily disabled with xspecify'

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I think I'd also like to see an example method which is skip which would function just like the RSpec::Core::ExampleGroup.pending did in 2.x. IMO, xit, xspecify, etc are handy short cuts for when you want to temporarily skip an example (only one letter different!) but are not particularly descriptive. skip is much more descriptive about what it does, so for situations where you actually want to commit an example as skipped, the more descriptive skip form would probably be better/more intention revealing.

lambda { v.call(a) }
end
stack.call
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I'm trying to wrap my head around why this new pending_executors thing is needed but I'm not getting it. Can you explain? (Maybe in a committed code comment rather than as a github reply is better so we can refer to it in the future).

This comment has been minimized.

Copy link
@JonRowe

JonRowe Feb 1, 2014

Member

One of the other PR's @xaviershay has open utilises it to extend this.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I know, I saw that. I'm trying to understand why this wasn't needed before but it is now. That's the piece I don't understand.

@xaviershay -- I'm online now and on IRC. If it's easier to explain over IRC feel free to pop in there.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

This have never worked properly. It's required now because some specs in rspec-mocks are marked as pending so previously have never been run. Without this change they actually pass and are incorrectly failed as "fixed".

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

So it has worked in most situations for years. David initially fixed this in 644e372 by adding a call to teardown_mocks_for_rspec from within the pending logic here. There's cukes there showing it work with each of the mock framework adapters.

When I changed the rspec-mocks lifecycle to have the nested/stacked space support, the extra call to teardown_mocks_for_rspec caused a problem because it meant that the setup/teardown calls were no longer balanced. I fixed that in 6661803, which attempts to translate a failure into marking :pending_fixed as false. That works for all the examples we have testing this from within rspec-core but there's an edge case I didn't think of that has now been uncovered.

The newly uncovered edge case is here in rspec-mocks:

https://github.com/rspec/rspec-mocks/blob/7b9930ba946e288c095cf9ebd3ed27d3820edb65/spec/rspec/mocks/matchers/receive_message_chain_spec.rb#L63-L73

The problem is that there are 2 separate errors that are happening. On the and_call_original line, it is raising this error:

#<RSpec::Mocks::MockExpectationError: Double is a pure test double. `and_call_original` is only available on a partial double.>

This causes the pending block to immediately abort and set :pending_fixed to false. Later, the mock object verification happens, and a second error is raised. As a result, when we deal with this here, pending_fixed is already false, so it doesn't deal with it properly.

Your solution fixes this problem, but has two downsides to it:

  • It only works for rspec-mocks, not for other mock framework adapters. While rspec-mocks is the preferred/recommended mocking library to use with rspec-core, we want to maintain feature/behavior parity as much as possible, and so we need a solution that works with other mocking libraries.
  • There's the code that's meant to address this that I added in 6661803. I don't want us to maintain two separate ways to deal with this.

I think it should be doable to update the rescue logic in 6661803 to handle this case. As a side benefit, if you fix this directly in rspec-core w/o the extra rspec-mocks PR needed, there's no PR ordering dependency :).

Let me know if you have questions.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

BTW, thanks for explaining -- the bit about the failure being in the rspec-mocks specs gave me what I needed to dig in and figure out why what we had before wasn't working :).

@xaviershay
Copy link
Member Author

xaviershay commented Feb 1, 2014

Thanks for all the comments! Pretty sure I know how to address most of them, working on it today.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 1, 2014

See new commits, I think they address everything.

I just rebased this and @myronmarston's new alias_example_group_to feature is failing with these changes, so need to investigate that some more.

#
# @see RSpec::Core::Pending#pending
def pending(*all_args, &block)
desc, *args = *all_args

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

Why not just define this as:

def pending(desc, *args, &block)
  # ...
end

...rather than pending(*all_args) and then split it on the first line?

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

needs a default of nil, but yeah that works.

callback = Proc.new { pending(reason) { instance_exec(&block) } }

examples << RSpec::Core::Example.new(self, desc, options, callback)
examples.last

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

There's a fair bit of duplication between this and the define_method within define_example_method. I'm concerned about future maintainability as the shared bits will need to evolve together. Here's an alternate way that I think will be more maintainable:

def self.define_example_method(name, extra_options={})
  define_method(name) do |*all_args, &block|
    desc, *args = *all_args
    options = Metadata.build_hash_from(args)
    options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
    options.update(extra_options)

    if options[:pending]
      options, block = ExampleGroup.pending_metadata_and_block_for(options, block)
    end

    examples << RSpec::Core::Example.new(self, desc, options, block)
    examples.last
  end
end

def self.pending_metadata_and_block_for(options, block)
  if String === options[:pending]
    reason = options[:pending]
  else
    options = options.merge(:pending => true)
    reason = RSpec::Core::Pending::NO_REASON_GIVEN
  end

  callback = Proc.new { pending(reason, &block) } # I thikn you can forward the block to pending w/o instance_exec
  return options, callback
end

define_example_method :pending, :pending => RSpec::Core::Pending::NO_REASON_GIVEN

Basically, this inverts it: rather than making :pending => true metadata delegate to a pending class method, it replaces the metadata and block for :pending => true and makes pending just like any other example method alias, where it just adds :pending metadata.

The previously duplicated parts are no longer duplicated here.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 1, 2014

Author Member

good spot! Changed in most recent commit.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 1, 2014

Build failures are the same expected RSpec mock errors.

@@ -888,26 +888,55 @@ def define_and_run_group(define_outer_example = false)
end
end

%w[pending xit xspecify xexample].each do |method_name|
describe "::pending" do

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I've mentioned this elsewhere, but I'm not a fan of using :: for message sends. I know you're being consistent, but my preference is to change all the places we work in to use the more standard . for message sends. Mind updating it?

before do
@group = ExampleGroup.describe
@group.pending { fail }
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 1, 2014

Member

I'm generally not a fan of before hooks and ivars. ivars spring into existence when referenced, which means you'll get a NoMethodError for nil if the ivar is renamed or typod. (Or, in the pathological case, you could get a false positive for an expectation like expect(@some_mispelled_ivar).to be_nil, where it's only nil because you misspelled it, not because the @foo = Foo.foo line in before returned nil like you expected).

Instead I would tend to favor let for this:

let(:group) { ExampleGroup.describe { pending { fail } } }

it 'generates a pending example' do
  group.run
  expect(group.examples.first).to be_pending
end

# ...

Any reason you chose the before/ivar route?

@xaviershay
Copy link
Member Author

xaviershay commented Feb 2, 2014

See last 4 commits. The implementation of new pending behaviour feels ... uncomfortable, possibly just because this code felt uncomfortable to start with and I haven't done any work it in before. I don't totally have all the begin/rescue blocks and different states in my head...

@xaviershay
Copy link
Member Author

xaviershay commented Feb 2, 2014

Turns out changing the behaviour of pending inside an example also accidentally fixed the rspec mocks problem! I added some explicit spec coverage anyways, and removed the pending executor stuff.

I think all feedback has been addressed now. There is the open issue of whether we want to provide a different interface to formatters, since that won't be a backwards incompatible change I don't think it needs to be included here.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

pending.rb and example.rb need another review, I changed them pretty significantly. The logic is a bit clearer now.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

urgh actually hold off, still a few more outstanding issues.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

This is still a bit of mess when interacting with mocks, but for the most part you just get specs marked as pending.

See 41ab9a7#diff-3bac19fc9abdbdd90aaec18fabf14a95R211 for my current understanding of the problem.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

@myronmarston
Copy link
Member

myronmarston commented Feb 9, 2014

Does anyone have a dev environment already set up that lets them easily regenerated all the HTML formatter fixtures? I'm not feeling motivated to install all the required rubies... Need to run the following:

I think I can do this. I'll take a stab at it later tonight (I'm only online for a couple minutes right now).

We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot.

I'm personally a fan of this feature and it's never been a maintenance burden before. It also seems like the kind of thing that would be hard to achieve in an extension gem using public APIs. I can understand your frustration, though...I'll try to take a look at things later tonight to see where things stand.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

I'm mostly worried about shipping an inconsistent feature. Although it's inconsistent now...

Actually, if we remove the pending-with-a-block-in-an-example feature, I think the problem goes away. You can still use skip with :if to conditionally evaluate blocks. You still get execute and fail on pass behaviour from either normal pending keyword or pending at example group scope. Thoughts?

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

Semantically, saying "this part of the spec is broken now, and should work in the future, but the rest of the spec still works fine and is valuable" seems unlikely and confusing to me. Instead, marking it pending halfway through (without a block) seems a lot clearer and easy to reason about.

Even in the case of an acceptance spec, executing a pending block half-way through the spec is problematic since it puts you in an unknown state.

To put it another way, I think the high-level functionality that used to be provide by pending-with-a-block ("let me know when this passes") is useful, but the old API for it is no good. By keeping the functionality but with the new API, and removing the old API, we get a much better product.

@myronmarston
Copy link
Member

myronmarston commented Feb 9, 2014

Ah, I misunderstood what you were talking about. I thought you were talking about the feature as a whole, and not simply the "pending block from within an example" aspect (which was the only API to the feature in 2.x). Now that I get what you're saying, I think I agree: it doesn't seem needed or useful, and the new API seems much better.

Actually, there is one case where it might still be useful: for code that is being written against both the RSpec 2.x API and RSpec 3.x API. Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use. That's probably not a big deal though: I can't think of any rspec extension gems that use this (as it's generally something in a specific end-user spec, not in an extension gem). It might affect Sequel, though, as I know it's suite is meant to work on RSpec 1.x, 2.x and 3.x (see jeremyevans/sequel@101cc3a).

I don't think that's a strong enough argument for it, though....so my vote is to axe it.

@markijbema
Copy link

markijbema commented Feb 9, 2014

Noticed that you have to do some effort to generate those fixtures locally. Is this something which needs to be done a lot? If so, you could consider leveraging Travis to generate them for you.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

I thought you were talking about the feature as a whole

Initially I was, but I changed :)

Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use.

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible. (In 2.9 this case will deprecate and recommend a change to skip or no block.)

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

@markijbema This would be the second time I've had to do it. They change pretty rarely. I really just need to get my dev environment set up properly...

@grddev
Copy link

grddev commented Feb 9, 2014

As I said earlier, my only strong opinion is that there should be a forward-compatible transition from RSpec 2.x, that:

  • Has the new behaviour
  • Will continue to work in 3.x
  • Does not produce any deprecation warnings in 2.99

Cutting the pending block feature seems to remove this possibility, or have I misunderstood something?

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

@grddev you can either switch to skip with a block, or just remove the block from your pending call. 2.9 will suggest both of these options.

@markijbema
Copy link

markijbema commented Feb 9, 2014

@grddev I understand what you want, and I agree that it is preferable, but there isn't really a way if we want to call this feature 'pending' in 3, right? Is there a way rspec can differentiate between a pending with block which was meant to be run, and fail if succeed, and a block which was meant not to be run? The only solution I see which would be a little closer is allow something like:

pending "foo", run: true do
end

which shows that it should be run in 2.99, and have the run param be optional in 3.

Would it be an idea to make the pending inside the block deprecated from version 3 onward? I don't think it has much more value above this syntax, right? I think that would solve the transition, since then you could transition in 2.99, and transition in 3 again. It's not perfect though. I don't see how you could solve this with one version though. For a gem I maintain we did some sequential upgrades to work around problems like this (reappropriating existing methods/names). You could also do it with 2.98 and 2.99 of course, but it feels a bit like a kludge.

@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

HTML fixtures need re-gen ... am hoping to be able to rebase this on top of #1306 to solve that problem.

@myronmarston
Copy link
Member

myronmarston commented Feb 9, 2014

If we remove it completely, then pending { whatever } will still "work", it will mark the example as pending without running the block. That doesn't seem terrible.

Might be worth warning and/or raising an error if a block is given to pending since it could be a common mistake from people not ware of the changes who are used to 2.x behavior, but I agree, either way, not terrible.

* Execute pending examples and mark as failed if they succeed.
* Removed pending with a block.
* Introduce `skip` method and metadata for old pending behaviour.

This is a backwards-incompatible change.

Implements #1208.
@xaviershay
Copy link
Member Author

xaviershay commented Feb 9, 2014

I'm done making changes to this, please give it a final once over.

Might be worth warning and/or raising an error if a block is given to pending.

We're already providing a warning in 2.99, I don't think it's worth the code here. In the common case you'll get flagged anyway, since the block won't execute and the example will pass, causing a failure.

an example is skipped using xexample
# Temporarily skipped with xexample
# ./temporarily_skipped_spec.rb:8
"""

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

There's a 5th way: tagging an example with :skip => "reason why". Not a merge blocker.


raise Pending::PendingExampleFixedError,
'Expected example to fail since it is pending, but it passed.',
metadata[:caller]

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

Interesting...I didn't know you could pass a different backtrace to an error like this. It's probably more user friendly to pass the metadata caller like this, but it's also potentially deceptive: if the user wants to actually see where the error was raised, (so they can look into the rspec code that raised it), the backtrace won't tell them like it usually would.

I've actually been thinking of doing away with metadata[:caller] -- as you discovered, caller is surprisingly slow, and, AFAIK, it's only needed to support line number filtering, but ruby 1.9+ provides us with a much more efficient way to support that: Proc#source_location gives us the file and line number where a block originates, and is much cheaper than using caller, so I'm planning to try to move that direction after beta2 ships, at which point it won't be available to pass here anymore. Ruby 1.8.7 would still use caller but it would just be used to set metadata[:source_location] or something similar and there would no longer be a :caller key in the metadata.

How important to you is it that metadata[:caller] is passed here?

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 10, 2014

Author Member

source_location would be fine. Without this, you have no idea where your pending block is in the error output.

metadata[:execution_result][:pending_exception] = e
else
set_exception(e)
end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

There are a few other places that call set_exception and in the cases that trigger those calls, if the example is pending, should it also store the exception to metadata[:execution_result][:pending_exception] = e? Maybe the pending? check should be moved directly into set_exception? Seems like anytime set an exception we'd want this treatment, not just here, right? (Might be worth adding a spec for one or more of those cases if there is indeed a bug).

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 17, 2014

Author Member

yeah this is a bug, fix on the way.

# @overload skip(message, &block)
#
# Marks an example as pending and skips execution when called without a
# block. When called with a block, skips just that block and does not

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

I'm trying to understand what "skips just that block" means. Take this for example:

it "does three things" do
  do_thing_1
  skip { do_thing_2 }
  do_thing_3
end

Saying it "skips just that block" suggests this is equivalent to the following:

it "does three things" do
  do_thing_1
  # do_thing_2
  do_thing_3
end

...so I'm not sure I see the point of supporting blocks at all -- IMO it makes even less semantic sense than pending with a block.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 10, 2014

Author Member

You're right, it's kind of useless. I mention this in the YARD doc, it's provided as an automatic upgrade from 2.99 (so you can just switch it in from pending), but isn't really useful for new stuff.

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

I still don't get what passing a block does...is it the same as commenting out the code within the block?

What I'm confused about is that I thought that "skip" is a property of the example so I don't understand how it can even apply to a block of code within an example without applying to the whole example.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 17, 2014

Author Member

is it the same as commenting out the code within the block

yes

What I'd like to do is deprecate it post-3.0 and then remove it. We could remove it now, just means transpec would need to convert:

it  do
  pending {
    # blah
  }
end

to

it do
  skip
  # blah
end

Hrmmm that's probably fine actually, then we could remove it now.

This comment has been minimized.

Copy link
@xaviershay

xaviershay Feb 17, 2014

Author Member

oh I remember why, I wanted an automatic replacement for:

it do
  pending(:if => some_predicate) {
    # Code A
  }
  # Code B
end

Could just convert that to the closest semantic equivalent though (note in the above snipped code B would not be executed if the block was pending, which is weird).

  pending if some_predicate
  # code A
  # code B
end
# fail
# end
# end
# end

This comment has been minimized.

Copy link
@myronmarston

myronmarston Feb 10, 2014

Member

BTW, thanks for putting the effort into doc'ing things so well!

@myronmarston
Copy link
Member

myronmarston commented Feb 10, 2014

Thanks for working so hard on this, @xaviershay. I left a few more comments, but any of them that are worth addressing can be addressed in a separate PR. I'm 100% OK with this being merged as is.

@markijbema
Copy link

markijbema commented Feb 10, 2014

Thanks!

@xaviershay
Copy link
Member Author

xaviershay commented Feb 10, 2014

Merging. Will investigate potential bugs around the set_exception, but I'd prefer to get this in beta2 to get some real usage out of this.

xaviershay added a commit that referenced this pull request Feb 10, 2014
Mark pending blocks as failed if they succeed.
@xaviershay xaviershay merged commit facc88c into master Feb 10, 2014
1 check passed
1 check passed
default The Travis CI build passed
Details
@xaviershay xaviershay mentioned this pull request Feb 17, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.