Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Shared Examples: context prevents "it" blocks from being executed when trying to execute them using a line number #835

Closed
jmuheim opened this Issue · 20 comments

8 participants

@jmuheim

Just noticed that my shared examples' it blocks were not executed when I ran them with an exact line number like so:

rspec spec/my_spec.rb:123

But when I ran them as the whole file, the it blocks were executed as expected:

rspec spec/my_spec.rb

After some debugging, I found that my shared examples' it blocks were in a context block - when I removed them, I could successfully execute them using a line number.

Is this a known bug/feature? Should I avoid contexts in shared examples?

@myronmarston

This is not a known bug or feature, but in general, the line number filtering of examples from a shared example group is a bit wonky, as you reported in #793.

@jmuheim

Yes, it's not the line number of the shared example, I know that I can't start it using this one. I started it using the spec's file number where I call the it_behaves_like method. So it definitely should work, but it doesn't when there's a context block surrounding the it blocks. Should I try to give you a concrete example?

@jithugopal

@sientia-jmu On an unrelated note, for shared examples I leave the comfort of my text editor and run specs with the '-e' flag.

In spec/models/frog_spec.rb

it_behaves_like "croakable", Frog

In spec/support/croakable.rb,

shared_examples "croakable" do |model_name|
  it "croaks in that irritating voice" do
    ...
  end
end
$> rspec -e "croaks in that irritating voice" spec/models/frog_spec.rb
@jmuheim

Thanks, @jithugopal , this is a very useful hint.

Still, it doesn't answer my question about whether contexts must be avoided in shared examples, or not.

@dchelimsky
Owner

@sientia-jmu no, you should not have to avoid nested groups (i.e. describe or context) in shared groups.

@soulcutter
Collaborator

A small (but complete) example would be much-appreciated.

@pstare

Has this been fixed yet? If not, I can provide a concrete example, but it really shouldn't be necessary -- the issue description is quite clear.

If you run a shared example by invoking the line in a spec file where the

it_behaves_like "the shared example group"

is defined, anything in the shared example inside a context block does not get executed.

i.e.

rspec -fd spec/lib/whatever_spec.rb:117

Will only run the it blocks that are not nested inside a context or describe.

@dchelimsky
Owner

@pstare a concrete example always helps because people might misinterpret descriptions. Here is an example that I think follows the description, but it works perfectly fine:

shared_examples "thing" do
  context "in some context" do # this is line 2
    it "does something" do # this is line 3
    end
  end
end

describe "something" do
  it_behaves_like "thing" # this is line 9
end
$ rspec example_spec.rb:2 -fd
Run options: include {:locations=>{"./example_spec.rb"=>[2]}}

something
  behaves like thing
    in some context
      does something

Finished in 0.00069 seconds
1 example, 0 failures

$ rspec example_spec.rb:3 -fd
Run options: include {:locations=>{"./example_spec.rb"=>[3]}}

something
  behaves like thing
    in some context
      does something

Finished in 0.00079 seconds
1 example, 0 failures

$ rspec example_spec.rb:9 -fd
Run options: include {:locations=>{"./example_spec.rb"=>[9]}}

something
  behaves like thing
    in some context
      does something

Finished in 0.00084 seconds

Obviously I'm missing something. Please help me understand what it is with a concrete example.

@pstare

@dchelimsky, sorry about that. In paring my spec files down to try to present the clearest example to you guys, I think I've found some key information. The expected behaviour is what I get when the shared example and the actual spec are in the same file, as you have above.

Here is my configuration:

file example_spec.rb:

require 'spec_helper'

describe "Top Level Spec Describe" do
  context "Spec Context" do
    it_behaves_like "something shared" # this is line 5
  end
end

file sample_shared_examples.rb: (this file is in spec/support, and all files in spec/support are required by a loop in my spec_helper.rb)

shared_examples_for "something shared" do
  specify "this will work for sure because it is not in a context or describe" do
    true.should be_true
  end

  context "Context within shared example group" do
    specify "this will not work when run with line numbers" do
      true.should be_true
    end
  end

  describe "Describe within shared example group" do
    specify "this will also not work when run with line numbers" do
      true.should be_true
    end
  end
end
$ bundle exec rspec -fd --color spec/lib/api/site/example_spec.rb -d --backtrace
spec_helper require loop, requiring: /Users/pstare/api_rspec/spec/support/sample_shared_examples.rb

Top Level Spec Describe
  Spec Context
    behaves like something shared
      this will work for sure because it is not in a context or describe
      Context within shared example group
        this will not work when run with line numbers
      Describe within shared example group
        this will also not work when run with line numbers

Finished in 0.00271 seconds
3 examples, 0 failures
$ bundle exec rspec -fd --color spec/lib/api/site/example_spec.rb:5 -d --backtrace
spec_helper require loop, requiring: /Users/pstare/api_rspec/spec/support/sample_shared_examples.rb
Run options: include {:locations=>{"./spec/lib/api/site/example_spec.rb"=>[5]}}

Top Level Spec Describe
  Spec Context
    behaves like something shared
      this will work for sure because it is not in a context or describe

Finished in 0.00224 seconds
1 example, 0 failures
[~/api_rspec]
$ ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [i686-darwin12.2.1]
[~/api_rspec]
$ bundle show | grep rspec
  * rspec (2.10.0)
  * rspec-core (2.10.1)
  * rspec-expectations (2.10.0)
  * rspec-mocks (2.10.1)
  * rspec_junit_formatter (0.1.1)
@pstare

I'm obviously not running the latest and greatest of anything for this project. Upgrading ruby is not really an option at this time; however, since someone reported this exact issue only 2 months ago, I am assuming the issue is still present in 1.9.3 and the latest rspec-core.

@dchelimsky
Owner

@pstare aha! That's a very important distinction (separate files). Thanks for the clarification. I'm not sure I'll have time to look at this for a couple of days. I'll assign it to myself if/when I do. In the mean time, @myronmarston, @alindeman, @JonRowe, @samphippen, @soulcutter - if any of you decide to look at this can you assign yourself so we don't dup the effort?

I am assuming the issue is still present in 1.9.3 and the latest rspec-core.

Generally (not 100% of the time - we're not perfect) you can assume it's still an issue if the github issue is still open.

@soulcutter soulcutter was assigned
@soulcutter
Collaborator

Nesting an example group inside a shared context changes the metadata[:example_group][:file_path] which results in it not passing through the location inclusion filter[1].

It seems that the assumption was that preceding declarations would always come from the same file. I suppose this is a corner case where that's a bad assumption... Figured I'd share these investigation results in case it sparks any ideas with other collaborators.

@myronmarston myronmarston added this to the 3.0 milestone
@xaviershay xaviershay referenced this issue from a commit
@xaviershay xaviershay Failing spec for filtering by location
Relates to #835

Conflicts:

	spec/spec_helper.rb
dd5b096
@xaviershay
Collaborator

My notes from investigating this just now:

  1. parent_example_group is used to apply the location filter against. For the top level shared context, this is the example group in which it_behaves_like as called. For the nested context, it is the top level shared context. This conceptually makes sense: we calculate a full description by working our way up this tree.
  2. parent_example_group is used also for location filtering, which as @soulcutter demonstrated doesn't work for this case.
  3. Given 1, I don't think we want to change the semantics of parent_example_group.
  4. From inspection of metadata, it doesn't look like we can distinguish between a shared context and not currently. (If we could, location filtering could recurse up to the first non-shared context.)
  5. Adding 4 should be an additive non-breaking change.
  6. Given 5, this should not be a release blocker.

I'm new to this part of the code, if 1-5 seem reasonable to others I can take a swing at a fix and maybe even get it in before 3.

I pushed up a branch that rebases @soulcutter's repro against latest master: https://github.com/rspec/rspec-core/tree/issue-835

@myronmarston
Owner

@xaviershay -- your assessment makes sense to me. I've changed the tag on this issue to "Not a release blocker". There are other filtering issues/improvements (such as #952 and #793) that we've pushed off until post 3.0 (hopefully 3.1) and this improvement could fit in with those quite nicely.

There is something I've been wanting to add to RSpec with regards to shared example groups that would probably help here: storing the stack of shared example groups in the metadata (e.g. when you have nested shared example groups). We can use this to print out a more complete "shared group backtrace" rather than just one level like we do here:

https://github.com/rspec/rspec-core/blob/e4e43d20a4d5988f200f40752a76c8582830d199/lib/rspec/core/formatters/base_text_formatter.rb#L232-L234

We can also use it to trace through the shared group inclusions in order to filter here.

@soulcutter
Collaborator

I realize this doesn't add too much to the conversation, but having spent time looking into this a while back, I agree with those investigation notes and Myron's probable solution. I was actually surprised that there isn't that sort of stack to begin with.

@benzittlau

Is there a target release for a fix to this? In my development flow executing subsets of tests is something I use frequently and is driven from my IDE, so this is a pretty awkward problem for me.

@myronmarston

Is there a target release for a fix to this?

Not yet. If it was slated for inclusion in a particular release, we'd add it to a milestone.

That said, I was thinking that I might work on the shared example group issues (this one, #1209 and #793) for 3.2.

@benzittlau

@myronmarston Thanks for the response. I'm a little confused though as it looks like this issue's currently assigned to milestone 3.0. Am I misreading something?

@myronmarston myronmarston removed this from the 3.0 milestone
@myronmarston

@myronmarston Thanks for the response. I'm a little confused though as it looks like this issue's currently assigned to milestone 3.0. Am I misreading something?

No. I forgot we had added it to the 3.0 milestone (as a nice-to-have for 3.0) but then it didn't make the cut.

@myronmarston myronmarston referenced this issue from a commit
@myronmarston myronmarston Fix location filtering.
For examples defined in an example group in a shared
example group in an external file, it did not work
properly because the location filtering only considered
the immediate parent example group’s file path, which
did not match the passed location filter. By considering
all parent example group file paths, it filters properly.

Fixes #835.

Note: this commit began from a failing spec that
originated with @soulcutter and was fixed up by
@xaviershay.
10791a2
@myronmarston
Owner

Sorry for taking so long to fix this, but I have a fix for it now in #1837 if anyone wants to try it out.

@JonRowe JonRowe closed this in #1837
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.