Skip to content

A more minimal approach to retrying failed examples #596

Closed
wants to merge 7 commits into from

7 participants

@antifun
antifun commented Apr 6, 2012

So after the discussion on #456, here's a crack at a less-invasive way to do the retry.

Given a formatter that outputs the names of failed examples only like so:

-e 'Example name that failed'
-e 'Another example name that failed'

You can get that output into a file using an rspec invocation like so:

rspec specs/*.spec --format f -o failures.txt ...

And then upon determining that some examples have failed, rerun just those examples like so:

rspec specs/*.spec -O failures.txt ...

In order to make this possible I had to make two changes to rspec itself, not counting the formatter: -e is now additive, and options provided in the -O file (and in .rspec, etc.) are now split using shell semantics rather than on whitespace (which I would have considered a bug).

Let me know what you think. The formatter is perhaps controversial if only because it's so trivial. I favor just going ahead and making it work out of the box, but I could also be convinced of alternatives like splitting it out into a separate distribution or having it not output the "-e" hiccup and leaving the prepending of "-e" to each line in this usage to the enterprising rspec user.

antifun added some commits Mar 26, 2012
@antifun antifun Failed-examples-only formatter using example names.
Intended for use with -O; outputs "-e" followed by the full name of the
failed example.
cdb3a45
@antifun antifun Use shell semantics when trying to split "command-line" options.
A raw split causes carefully-escaped example names to get screwed up.
It's also more consistent with the spirit of the external options files
(equivalence with what you might provide on the command line).
164fb29
@antifun antifun Make the -e example name option additive.
It will now run examples whose names match any of the provided regular
expressions.
0a6dc7e
@antifun antifun Feature for rerunning failed examples.
Ties together the failure formatter, the additive -e, and use of the
supplemental -O command-line options file.
835f24d
@dblock dblock added a commit to dblock/rspec-core that referenced this pull request May 4, 2012
@dblock dblock Ported support for mutliple -e options from #596. 2f88804
@dblock
dblock commented May 4, 2012

I've pulled two parts from here into separate pull requests.

  • Fix for shell-parsing -e, #610
  • Cumulative -e options, #614
@dblock
dblock commented May 4, 2012

+1 for this, I think it gets the job done and I think it's generic enough - I would merge from upstream once #610 and #614 are in, so that we can see just the diff that accomplishes the retry logic

@dblock
dblock commented May 4, 2012

I think maybe the formatter should output into a well known file though, like rspec.failures, otherwise if it uses the file in -o, you can't use other formatters for console status, right?

@antifun
antifun commented May 8, 2012

Not sure I follow the last question. I think it works just like the other formatters, where a pair of --format and -o determine an auxiliary output, and a (trailing?) --format alone controls the console output.

The feature in 835f24d almost does this. I think if I'd left off the -o first_run.txt and just looked at the console output it would have worked the same way.

@dblock
dblock commented May 8, 2012

Maybe I am missing something, but --format is cumulative, while -o applies to all formatters using output. So what does --format f --format d --o output.txt do? I think it will put both outputs into output.txt. Which means that to use the f formatter you cannot have any other console output?

@antifun antifun Rework retry feature to more clearly demonstrate functionality.
Split the feature to hopefully assuage dblock's concern about how
multiple output/format specifications work.
d0a6da6
@travisbot

This pull request fails (merged d0a6da6 into ba9b62d).

@travisbot

This pull request passes (merged fbbd351 into a0202ea).

@dblock
dblock commented May 16, 2012

While this implementation is being debated, we pulled #610 and #614 and used this idea (and roughly this code) to do this outside of RSpec proper. Blogged http://artsy.github.com/blog/2012/05/15/how-to-organize-over-3000-rspec-specs-and-retry-test-failures/.

@antifun
antifun commented May 16, 2012

Thanks for that blog. I think I'll probably just leave well enough alone and do it outside of rspec as well. You're doing something slightly fancier than what we are, but don't mind if I borrow some inspiration from your example! :-)

@oggy
oggy commented May 17, 2012

FWIW, I've pushed another solution -- a gem which wraps the rspec command which does not require code changes to the project under test -- here: http://github.com/oggy/respec .

Will make this use -e once pull #614 lands in stable.

@justinko justinko closed this May 17, 2012
@TylerRick

I'd be in favor of including this formatter in core. Users coming from cucumber are going to look for and expect an equivalent to cucumber's rerun format, myself included...

Now that #610 and #614 have been merged into master, it looks like the only thing still needed for this feature to be available is the failures formatter.

Any chance of seeing this final small piece merged in as well?

Otherwise, we should extract FailuresFormatter (and associated feature files) into a nice reusable gem and post a link to it here...

I don't really like the idea of using a completely separate respec command when all we need is this formatter and the -O option.

@dblock's blog post was valuable in showing how to pull everything together using rake tasks (and makes me think we should suggest adding a spec:rerun task to rspec-rails), but I don't want to have to copy and paste the formatter into each project...

@dblock
dblock commented Aug 14, 2012

Here we go: https://github.com/dblock/rspec-rerun

The formatter is included, with specs, I'm working on a README and stuff like that. It needs built-in rake tasks to do retry s well, please contribute.

@dblock
dblock commented Aug 14, 2012

... and it turned out to be simpler than I thought.

Released rspec-rerun 0.1.0. To use, just add it to Gemfile and change the default task in Rakefile:

require 'rspec-rerun'
task :default => "rspec-rerun:spec"
@dblock dblock referenced this pull request in dblock/rspec-rerun Feb 6, 2013
Closed

A bit unsure about the intended workflow #6

@aprescott

Check out #456 (comment) for another option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.