Skip to content

A more minimal approach to retrying failed examples #596

wants to merge 7 commits into from

7 participants

antifun commented Apr 6, 2012

So after the discussion on #456, here's a crack at a less-invasive way to do the retry.

Given a formatter that outputs the names of failed examples only like so:

-e 'Example name that failed'
-e 'Another example name that failed'

You can get that output into a file using an rspec invocation like so:

rspec specs/*.spec --format f -o failures.txt ...

And then upon determining that some examples have failed, rerun just those examples like so:

rspec specs/*.spec -O failures.txt ...

In order to make this possible I had to make two changes to rspec itself, not counting the formatter: -e is now additive, and options provided in the -O file (and in .rspec, etc.) are now split using shell semantics rather than on whitespace (which I would have considered a bug).

Let me know what you think. The formatter is perhaps controversial if only because it's so trivial. I favor just going ahead and making it work out of the box, but I could also be convinced of alternatives like splitting it out into a separate distribution or having it not output the "-e" hiccup and leaving the prepending of "-e" to each line in this usage to the enterprising rspec user.

antifun added some commits Mar 26, 2012
@antifun antifun Failed-examples-only formatter using example names.
Intended for use with -O; outputs "-e" followed by the full name of the
failed example.
@antifun antifun Use shell semantics when trying to split "command-line" options.
A raw split causes carefully-escaped example names to get screwed up.
It's also more consistent with the spirit of the external options files
(equivalence with what you might provide on the command line).
@antifun antifun Make the -e example name option additive.
It will now run examples whose names match any of the provided regular
@antifun antifun Feature for rerunning failed examples.
Ties together the failure formatter, the additive -e, and use of the
supplemental -O command-line options file.
@dblock dblock added a commit to dblock/rspec-core that referenced this pull request May 4, 2012
@dblock dblock Ported support for mutliple -e options from #596. 2f88804
dblock commented May 4, 2012

I've pulled two parts from here into separate pull requests.

  • Fix for shell-parsing -e, #610
  • Cumulative -e options, #614
dblock commented May 4, 2012

+1 for this, I think it gets the job done and I think it's generic enough - I would merge from upstream once #610 and #614 are in, so that we can see just the diff that accomplishes the retry logic

dblock commented May 4, 2012

I think maybe the formatter should output into a well known file though, like rspec.failures, otherwise if it uses the file in -o, you can't use other formatters for console status, right?

antifun commented May 8, 2012

Not sure I follow the last question. I think it works just like the other formatters, where a pair of --format and -o determine an auxiliary output, and a (trailing?) --format alone controls the console output.

The feature in 835f24d almost does this. I think if I'd left off the -o first_run.txt and just looked at the console output it would have worked the same way.

dblock commented May 8, 2012

Maybe I am missing something, but --format is cumulative, while -o applies to all formatters using output. So what does --format f --format d --o output.txt do? I think it will put both outputs into output.txt. Which means that to use the f formatter you cannot have any other console output?

@antifun antifun Rework retry feature to more clearly demonstrate functionality.
Split the feature to hopefully assuage dblock's concern about how
multiple output/format specifications work.

This pull request fails (merged d0a6da6 into ba9b62d).


This pull request passes (merged fbbd351 into a0202ea).

dblock commented May 16, 2012

While this implementation is being debated, we pulled #610 and #614 and used this idea (and roughly this code) to do this outside of RSpec proper. Blogged

antifun commented May 16, 2012

Thanks for that blog. I think I'll probably just leave well enough alone and do it outside of rspec as well. You're doing something slightly fancier than what we are, but don't mind if I borrow some inspiration from your example! :-)

oggy commented May 17, 2012

FWIW, I've pushed another solution -- a gem which wraps the rspec command which does not require code changes to the project under test -- here: .

Will make this use -e once pull #614 lands in stable.

@justinko justinko closed this May 17, 2012

I'd be in favor of including this formatter in core. Users coming from cucumber are going to look for and expect an equivalent to cucumber's rerun format, myself included...

Now that #610 and #614 have been merged into master, it looks like the only thing still needed for this feature to be available is the failures formatter.

Any chance of seeing this final small piece merged in as well?

Otherwise, we should extract FailuresFormatter (and associated feature files) into a nice reusable gem and post a link to it here...

I don't really like the idea of using a completely separate respec command when all we need is this formatter and the -O option.

@dblock's blog post was valuable in showing how to pull everything together using rake tasks (and makes me think we should suggest adding a spec:rerun task to rspec-rails), but I don't want to have to copy and paste the formatter into each project...

dblock commented Aug 14, 2012

Here we go:

The formatter is included, with specs, I'm working on a README and stuff like that. It needs built-in rake tasks to do retry s well, please contribute.

dblock commented Aug 14, 2012

... and it turned out to be simpler than I thought.

Released rspec-rerun 0.1.0. To use, just add it to Gemfile and change the default task in Rakefile:

require 'rspec-rerun'
task :default => "rspec-rerun:spec"
@dblock dblock referenced this pull request in dblock/rspec-rerun Feb 6, 2013

A bit unsure about the intended workflow #6


Check out #456 (comment) for another option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.