Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Summary output of shared failed examples is useless when examples defined in external file #793

Closed
jmuheim opened this Issue · 16 comments

4 participants

@jmuheim

When shared examples (that were defined in an external file, e.g. spec/support/breadcrumbs.rb) fail, their summary provides (to me) not the expected output.

The line references point to the shared examples' file, but it would be more useful if they would point to the xxx_spec.rb file from which they were invoked. This way I could just copy&paste the line reference and rerun it, but as it currently is, I have to search in the output before the summary to really see what examples didn't run.

Failed examples:

rspec ./spec/support/breadcrumbs.rb:6 # Breadcrumbs logged in as admin user Users behaves like a page with #index breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:16 # Breadcrumbs logged in as admin user Users behaves like a page with #new breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:28 # Breadcrumbs logged in as admin user Users behaves like a page with #create breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:45 # Breadcrumbs logged in as admin user Users behaves like a page with #show breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:59 # Breadcrumbs logged in as admin user Users behaves like a page with #edit breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:75 # Breadcrumbs logged in as admin user Users behaves like a page with #update breadcrumbs shows the expected breadcrumbs

Also, the very useful gem https://github.com/oggy/respec (which let's you re-run failed examples using respec f) uses the line reference to remember failed examples. This works perfectly for "normal" examples, but for shared examples, it's useless.

Content of .rspec_failures:

./spec/support/breadcrumbs.rb:6
./spec/support/breadcrumbs.rb:16
./spec/support/breadcrumbs.rb:28
./spec/support/breadcrumbs.rb:45
./spec/support/breadcrumbs.rb:59
./spec/support/breadcrumbs.rb:75

Is there a way to fix this? Or am I doing something wrong? It's convenient to have shared examples in their own external files in spec/support so they are loaded automatically before any other test is. Thank you.

@jmuheim jmuheim referenced this issue in oggy/respec
Closed

Doesn't work with shared examples #2

@jmuheim

I have created a fix for respec.

https://github.com/oggy/respec/pull/3/files

The formatter now looks through the example.metadata hash until it finds the first location that is a ..._spec.rb file. This is used, instead of simply the first location.

Maybe it would make sense to apply a mechanism like this for the default formatter, too?

@dchelimsky
Owner

Seems like a reasonable approach, however it should probably be based on RSpec.configuration.pattern so it works even when users change that to something like "**/*.spec".

@jmuheim

RSpec.configuration.pattern ? Can you explain this a bit further? Maybe with pseudo code. :+1:

@dchelimsky
Owner

See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration#pattern-instance_method

RSpec treats files matching this glob pattern as spec files. The default is "spec/**/*_spec.rb", but you can override that via RSpec.configure or the command line --pattern option, so rather than using a literal as you currently do, it'd be better to use the value of RSpec.configure.pattern.

@jmuheim

Ah I get it now! I'll apply this.

Is there a chance that something like this will find its way into the RSpec core? So all formatters would print the xxx_spec.rb line even for shared examples?

@dchelimsky
Owner

I think it's a good idea. @myronmarston WDYT?

@jmuheim

Hm any idea on how to create a regexp from RSpec.configuration.pattern?

@dchelimsky
Owner

Ugh. Good point. /me puts on thinking cap ...

@myronmarston

This has bugged me before as well, but I hadn't really given thought to how we might change it....so I'm glad to see this discussion taking place. A few thoughts:

  • It's also bugged me that the "shared example group called from" output is limited to one level. In cases where you have nested shared example groups, it'd be nice if the whole "shared example call stack" was displayed.
  • Once we get that call stack in place exposed from the metadata, this becomes far easier--you can just walk up that call stack to the original entry point to shared examples (e.g. the top-level it_behaves_like in a spec file).
  • That said, bear in mind that if you re-run the specs passing that line number, it'll run all of the shared examples, and not just the failing one.

Is there a chance that something like this will find its way into the RSpec core? So all formatters would print the xxx_spec.rb line even for shared examples?

Yep, I'd like to fix things so that any formatters which use dump_commands_to_rerun_failed_examples include the originating line for the use of the shared example group.

@jmuheim

I'm happy to see that I could start a serious discussion about this. Let me know if I can be of any more help. :+1:

@jmuheim

Is there anything going on about this topic?

@myronmarston
Owner

Not yet. It's still an open issue to fix. I've had more important things on my radar.

@jmuheim jmuheim referenced this issue in oggy/respec
Merged

Support for shared examples #3

@jmuheim

Wow, that's a fast answer! :+1:

Sure thing there is more important stuff to do. And don't forget to take some time to relax during weekend! :wink:

@jmuheim

@dchelimsky:

Hm any idea on how to create a regexp from RSpec.configuration.pattern?

Our friend Oggy created something that should do it. What do you think about it?
https://github.com/oggy/respec/blob/132d23f6265b7e36247aae7c055f334b763972a2/lib/respec/formatter.rb#L23

@dchelimsky
Owner

@sientia-jmu seems reasonable on first glance, but I'd have to dig deeper than I have time to right now to give you a real answer.

@myronmarston myronmarston added this to the Post 3.0 milestone
@GeekOnCoffee

Just ran into this, would love to get the discussion started again, see if we can't get this across the finish line

@JonRowe JonRowe closed this in #1835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.