Summary output of shared failed examples is useless when examples defined in external file

[jmuheim]

When shared examples (that were defined in an external file, e.g. spec/support/breadcrumbs.rb) fail, their summary provides (to me) not the expected output.

The line references point to the shared examples' file, but it would be more useful if they would point to the xxx_spec.rb file from which they were invoked. This way I could just copy&paste the line reference and rerun it, but as it currently is, I have to search in the output before the summary to really see what examples didn't run.

Failed examples:

rspec ./spec/support/breadcrumbs.rb:6 # Breadcrumbs logged in as admin user Users behaves like a page with #index breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:16 # Breadcrumbs logged in as admin user Users behaves like a page with #new breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:28 # Breadcrumbs logged in as admin user Users behaves like a page with #create breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:45 # Breadcrumbs logged in as admin user Users behaves like a page with #show breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:59 # Breadcrumbs logged in as admin user Users behaves like a page with #edit breadcrumbs shows the expected breadcrumbs
rspec ./spec/support/breadcrumbs.rb:75 # Breadcrumbs logged in as admin user Users behaves like a page with #update breadcrumbs shows the expected breadcrumbs

Also, the very useful gem https://github.com/oggy/respec (which let's you re-run failed examples using respec f) uses the line reference to remember failed examples. This works perfectly for "normal" examples, but for shared examples, it's useless.

Content of .rspec_failures:

./spec/support/breadcrumbs.rb:6
./spec/support/breadcrumbs.rb:16
./spec/support/breadcrumbs.rb:28
./spec/support/breadcrumbs.rb:45
./spec/support/breadcrumbs.rb:59
./spec/support/breadcrumbs.rb:75

Is there a way to fix this? Or am I doing something wrong? It's convenient to have shared examples in their own external files in spec/support so they are loaded automatically before any other test is. Thank you.

[jmuheim]

I have created a fix for respec.

https://github.com/oggy/respec/pull/3/files

The formatter now looks through the example.metadata hash until it finds the first location that is a ..._spec.rb file. This is used, instead of simply the first location.

Maybe it would make sense to apply a mechanism like this for the default formatter, too?

[dchelimsky]

Seems like a reasonable approach, however it should probably be based on RSpec.configuration.pattern so it works even when users change that to something like "**/*.spec".

[jmuheim]

RSpec.configuration.pattern ? Can you explain this a bit further? Maybe with pseudo code. 👍

[dchelimsky]

See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration#pattern-instance_method

RSpec treats files matching this glob pattern as spec files. The default is "spec/**/*_spec.rb", but you can override that via RSpec.configure or the command line --pattern option, so rather than using a literal as you currently do, it'd be better to use the value of RSpec.configure.pattern.

[jmuheim]

Ah I get it now! I'll apply this.

Is there a chance that something like this will find its way into the RSpec core? So all formatters would print the xxx_spec.rb line even for shared examples?

[dchelimsky]

I think it's a good idea. @myronmarston WDYT?

[jmuheim]

Hm any idea on how to create a regexp from RSpec.configuration.pattern?

[dchelimsky]

Ugh. Good point. /me puts on thinking cap ...

[myronmarston]

This has bugged me before as well, but I hadn't really given thought to how we might change it....so I'm glad to see this discussion taking place. A few thoughts:

• It's also bugged me that the "shared example group called from" output is limited to one level. In cases where you have nested shared example groups, it'd be nice if the whole "shared example call stack" was displayed.
• Once we get that call stack in place exposed from the metadata, this becomes far easier--you can just walk up that call stack to the original entry point to shared examples (e.g. the top-level it_behaves_like in a spec file).
• That said, bear in mind that if you re-run the specs passing that line number, it'll run all of the shared examples, and not just the failing one.

Is there a chance that something like this will find its way into the RSpec core? So all formatters would print the xxx_spec.rb line even for shared examples?

Yep, I'd like to fix things so that any formatters which use dump_commands_to_rerun_failed_examples include the originating line for the use of the shared example group.

[jmuheim]

I'm happy to see that I could start a serious discussion about this. Let me know if I can be of any more help. 👍

[jmuheim]

Is there anything going on about this topic?

[myronmarston]

Not yet. It's still an open issue to fix. I've had more important things on my radar.

[jmuheim]

Wow, that's a fast answer! 👍

Sure thing there is more important stuff to do. And don't forget to take some time to relax during weekend! 😉

[jmuheim]

@dchelimsky:

Hm any idea on how to create a regexp from RSpec.configuration.pattern?

Our friend Oggy created something that should do it. What do you think about it?
https://github.com/oggy/respec/blob/132d23f6265b7e36247aae7c055f334b763972a2/lib/respec/formatter.rb#L23

[dchelimsky]

@sientia-jmu seems reasonable on first glance, but I'd have to dig deeper than I have time to right now to give you a real answer.

[GeekOnCoffee]

Just ran into this, would love to get the discussion started again, see if we can't get this across the finish line