Option to re-run failing examples

[mcmire]

I'm not sure why no one has suggested this yet, but it would be nice if there were a way to do this. This comes in handy especially when you are running a file that has, say, 100 examples and 20 of them fail and you don't want to waste time re-running the other 80. Additionally, whatever is keeping track of failed examples should do so across multiple files that you may be running. So if I am running the full spec suite and some of them fail and they are in separate files, I want the ability to re-run just those.

So the way it'd work is that rspec would write the failed examples to a (non-anonymous) file (this could be a YAML file where the key is a filename and value is an array of line numbers). Given this file contains failed examples, if I then ran rspec --failed, RSpec should read the file and run only those examples, and if any succeed it removes them from the YAML file. If there are no examples left in the YAML file, then rspec --failed runs everything. So obviously running rspec with the option never overwrites the YAML file, but without the option rspec should always overwrite it, that way if you were to run it with the option right afterward you'd get a fresh file.

[justinko]

In my experience, when a suite fails, it is because of only a handful (usually one) of reasons. What that means is if you have 30 failing specs, chances are good that it is caused by one problem. So for me, I'll get the file and line number for one of those failing specs, make it pass, and then chances are very good that the other specs will pass. If that is not the case, that tells me the program has bad encapsulation.

Conclusion: the number of failed examples is not proportionate to the number of things you need to change/fix.

@myronmarston - has this been your experience as well?

[myronmarston]

That's often the case, but I still think this idea has merit. It sounds a lot like cucumber's rerun option. It would probably be good to align the API with cucumber's API for this.

[mcmire]

I would also agree that is generally true for unit tests, although I also have Capybara integration tests in my Rails app at work (which obviously take a lot longer to run than unit tests) for which that is less often true and for which this would probably be more useful.

I will see if I can take a look at how Cucumber does this and come up with a patch.

[dchelimsky]

We actually had this in rspec-1. My personal experience was that it worked the way I wanted it to most of the time, but when it didn't it took me a while to remember why. If we add this to rspec-2 I'd want the output to indicate that the specs being run were being constrained by the previous failures, and explain how to get around it.

[dcadenas]

+1

[oggy]

I'd love this too. In fact, just a built-in formatter which dumped the failing examples out to a file (like rspec 1.x) would make me real happy, as I tend to run rspec through my own wrapper anyway.

If such a formatter would be welcome I'd happily submit a patch. Otherwise I guess I'll do a gem.

[dchelimsky]

It's not enough to have a formatter if we're going to include it in rspec - we also need a means of running just the previously failed examples. I'd prefer this to be a command line option. If you want to submit a patch with both (formatter and command line option to run failed examples) I'd definitely consider it.

[nilclass]

This is how it works without a patch (I put it in my spec_helper.rb, in a rails project, but should work without rails as well, with a small adjustment to the first line):

    failure_log_path = Rails.root.join('rerun.txt')

    failed_examples = []

    if File.exist?(failure_log_path)
      rerunning = true
      failed_descriptions = File.readlines(failure_log_path).map(&:chomp)
      config.inclusion_filter = {
        :full_description => ->(desc) {
          failed_descriptions.any? { |d| desc =~ /#{d}/ }
        }
      }
    end

    config.after(:each) do |group|
      if group.example.instance_variable_get("@exception")
        failed_examples.push(group.example)
      end
    end

    config.after(:suite) do
      if failed_examples.any?
        File.open(failure_log_path, 'w') { |f|
          failed_examples.each do |example|
            f.puts(example.metadata[:full_description])
          end
        }
      elsif rerunning
        # no failures -> we (probably) fixed it!
        File.unlink(failure_log_path)
      end
    end

[oggy]

I'm happy to hear this. :)

I'm going to start on a patch. Here's what I'm thinking:

Simplest usage:

rspec --format=f|failures  # record the failures to the default path
rspec --failed             # rerun all recorded failures

The current directory is recorded along with the failing examples. If this directory no longer matches, you get a warning immediately, but the test run proceeds. This is to reduce confusion when you switch between multiple projects. If you've simply moved the project, however, you can just ignore the warning.

You can record failures to a project-specific file with a --failure-file=PATH option.

You can also specify the exact specs to rerun by passing --failed the numbers of the failures: --failed=1-5,7,13

I find the latter functionality useful when I need to insert a debugger at a point that many specs hit, but only in the context of a certain failing spec.

Thoughts?

@dchelimsky, would this eliminate the confusion you mentioned with rspec-1? Would you still want more than a single line of feedback that basically acknowledges you're in --failed mode?

[nilclass]

@oggy, does that mean that while recording failures, the output format (for stdout) cannot be specified any longer? In that case I would suggest going another way and maybe implementing a custom option (--record-failed or something) instead. While recording failures does produce output, it is not strictly a output format in the sense of "documentation", "progress" etc. which are meant to be a representation of results for users, not rspec itself. Any thoughts?

[oggy]

@nilclass: No. You can already specify multiple -f/--format options. And you can give each one their own -o/--out option.

For this reason, I'm actually inclined to use --out rather than --failure-file when writing the file for consistency with the other formatters. We'd still use --failure-file to specify a custom input failure file - i.e., when rerunning failed examples.

[mcmire]

@nilclass @oggy Good thoughts but why not always record failed examples all the time, why require a --format / --record-failed option? Here's my use case: oftentimes when I want to re-run failed examples, I have already run, say, a folder of examples and only after I do so do I realize I want to re-run the failed ones. If failed examples are always recorded, well hey look at that all I have to do is run rspec --failed and I've done just that, there's no need to go through the trouble of re-running all of them, including the ones I know are successful, when all I really want to do is re-run the ones that aren't. I think it's a more friendlier design to anticipate this. WDYT?

[oggy]

@mcmire, I dig that. Recording failures should be opt-out-able (e.g. you mightn't have write access to the default location), but otherwise I agree.

[dchelimsky]

I'd consider this, but the question then is where would the file live by convention.

One thing I've thought of is allowing for .rspec to become a directory. What is now .rspec would become .rspec/config and then we could have .rspec/failure_report.txt (or .yml). That would reduce the pollution of the project root and open the door for other file storage, like historical stats (which could be used for ordering as well - run the examples that fail most often first), etc. Obviously it would have to respect a .rspec file as well for compatibility with existing suites, but that wouldn't be difficult to do. It would just mean you couldn't use this feature until you changed the .rspec file to .rspec/config.

Thoughts?

/cc @myronmarston @justinko @patmaddox @spicycode

[nilclass]

@dchelimsky .rspec.d would do as well and be backwards compatible