So after the discussion on #456, here's a crack at a less-invasive way to do the retry.
Given a formatter that outputs the names of failed examples only like so:
-e 'Example name that failed'
-e 'Another example name that failed'
You can get that output into a file using an rspec invocation like so:
rspec specs/*.spec --format f -o failures.txt ...
And then upon determining that some examples have failed, rerun just those examples like so:
rspec specs/*.spec -O failures.txt ...
In order to make this possible I had to make two changes to rspec itself, not counting the formatter: -e is now additive, and options provided in the -O file (and in .rspec, etc.) are now split using shell semantics rather than on whitespace (which I would have considered a bug).
Let me know what you think. The formatter is perhaps controversial if only because it's so trivial. I favor just going ahead and making it work out of the box, but I could also be convinced of alternatives like splitting it out into a separate distribution or having it not output the "-e" hiccup and leaving the prepending of "-e" to each line in this usage to the enterprising rspec user.
Failed-examples-only formatter using example names.
Intended for use with -O; outputs "-e" followed by the full name of the
Use shell semantics when trying to split "command-line" options.
A raw split causes carefully-escaped example names to get screwed up.
It's also more consistent with the spirit of the external options files
(equivalence with what you might provide on the command line).
Make the -e example name option additive.
It will now run examples whose names match any of the provided regular
Feature for rerunning failed examples.
Ties together the failure formatter, the additive -e, and use of the
supplemental -O command-line options file.
Ported support for mutliple -e options from #596.
I've pulled two parts from here into separate pull requests.
+1 for this, I think it gets the job done and I think it's generic enough - I would merge from upstream once #610 and #614 are in, so that we can see just the diff that accomplishes the retry logic
I think maybe the formatter should output into a well known file though, like rspec.failures, otherwise if it uses the file in -o, you can't use other formatters for console status, right?
Not sure I follow the last question. I think it works just like the other formatters, where a pair of --format and -o determine an auxiliary output, and a (trailing?) --format alone controls the console output.
The feature in 835f24d almost does this. I think if I'd left off the -o first_run.txt and just looked at the console output it would have worked the same way.
Maybe I am missing something, but --format is cumulative, while -o applies to all formatters using output. So what does --format f --format d --o output.txt do? I think it will put both outputs into output.txt. Which means that to use the f formatter you cannot have any other console output?
--format f --format d --o output.txt
Rework retry feature to more clearly demonstrate functionality.
Split the feature to hopefully assuage dblock's concern about how
multiple output/format specifications work.
This pull request fails (merged d0a6da6 into ba9b62d).
Merge branch 'master' of https://github.com/rspec/rspec-core into off…
Change spec to reflect full_description being an array.
This pull request passes (merged fbbd351 into a0202ea).
While this implementation is being debated, we pulled #610 and #614 and used this idea (and roughly this code) to do this outside of RSpec proper. Blogged http://artsy.github.com/blog/2012/05/15/how-to-organize-over-3000-rspec-specs-and-retry-test-failures/.
Thanks for that blog. I think I'll probably just leave well enough alone and do it outside of rspec as well. You're doing something slightly fancier than what we are, but don't mind if I borrow some inspiration from your example! :-)
FWIW, I've pushed another solution -- a gem which wraps the rspec command which does not require code changes to the project under test -- here: http://github.com/oggy/respec .
Will make this use -e once pull #614 lands in stable.
I'd be in favor of including this formatter in core. Users coming from cucumber are going to look for and expect an equivalent to cucumber's rerun format, myself included...
Now that #610 and #614 have been merged into master, it looks like the only thing still needed for this feature to be available is the failures formatter.
Any chance of seeing this final small piece merged in as well?
Otherwise, we should extract FailuresFormatter (and associated feature files) into a nice reusable gem and post a link to it here...
I don't really like the idea of using a completely separate respec command when all we need is this formatter and the -O option.
@dblock's blog post was valuable in showing how to pull everything together using rake tasks (and makes me think we should suggest adding a spec:rerun task to rspec-rails), but I don't want to have to copy and paste the formatter into each project...
Here we go: https://github.com/dblock/rspec-rerun
The formatter is included, with specs, I'm working on a README and stuff like that. It needs built-in rake tasks to do retry s well, please contribute.
... and it turned out to be simpler than I thought.
Released rspec-rerun 0.1.0. To use, just add it to Gemfile and change the default task in Rakefile:
task :default => "rspec-rerun:spec"
Check out #456 (comment) for another option.