Runs specs while preserving only-failures

[christhekeele]

Hey!

There might be a way to accomplish this that I'm missing, but I'd like to be able to perform a spec run, capture the results for future --only-failures runs, but intermittently run other specs without overriding that failure log.

My usecase is operating on a large test suite that takes time to run to collect all the current failures, fixing them individually, and occasionally checking my progress against those failures.

The way I'm doing this today is by hacking an extra flag into my bin/rspec:

#!/usr/bin/env ruby
# bin/rspec

require 'fileutils'

if index = ARGV.index("--preserve-failures")
  failure_file = ENV.fetch('SPEC_FAILURES', "default_location")
  if File.exists?(failure_file)
    preserve_failure_file = failure_file + ".dup"
    FileUtils.cp(failure_file, preserve_failure_file)
    ENV['SPEC_FAILURES'] = preserve_failure_file
  end
  ARGV.delete_at(index)
end

#...

then later:

# spec/spec_helper.rb

RSpec.configure do |config|
  config.example_status_persistence_file_path = ENV.fetch('SPEC_FAILURES', "default_location")
  #...
end

This lets me run rspec to capture current failures, rspec --only-failures --preserve-failures to check in on how many I have made pass (without dismissing them in case I break them again in the process) and rspec --preserve-failures ./spec/specific_spec.rb to just see how I'm doing with the failure I'm currently working on.

This seems like it would be a nice feature to have supported in core if there isn't a supported way to do this already; I'm willing to work on it but wanted to take the temperature first.

[myronmarston]

My usecase is operating on a large test suite that takes time to run to collect all the current failures, fixing them individually, and occasionally checking my progress against those failures.

FWIW, this use case is what --next-failure is intended for: after running your suite and getting some failures, you can work through them individually by repeatedly running rspec --next-failure.

Anyhow, when you talk about "preserving failures", it's not totally clear to me what you mean. I can see a few different ways to understand what you're asking for:

1. When running a subset of a spec suite, preserve the status of any examples that did not run, so that examples that failed when they last ran are still marked as failed even if they were not included in the last run.
2. After running a suite, do not change the recorded status of any examples from "failed" to "pass" (e.g. preserving the record of the prior failures, plus recording any newly failing examples).
3. After running a suite, do not update the status file at all, preserving it exactly as it was before the current spec run.

I'm not sure which of these you are asking for. If it's the 1st one, well, RSpec already does that :). It's what makes --next-failure possible.

If it's the 2nd or 3rd cases: it seems to me that the situations where these are useful are pretty specialized, and I'm not sure there's a one-size-fits all approach to serving those cases. Some folks might want only the failures preserved, or all the current statuses preserved, or they might want a given set of failures "snapshotted" for later use (rather than simply preserving it for a specific spec run). Instead of having an option that supports a narrow, specialized case, I think people can manage this themselves by just making a copy of the status file when they don't want it updated, then copying it back when they want to re-use an old status file.

Maybe I'm missing something, though...

[christhekeele]

Nope, you've got it right--in particular, behaviour #2 is what I implemented in my snippets, pretty much verbatim copying the file around.

I agree it's pretty specific, for when you're playing whack-a-mole with a set of problematic tests that aren't sufficiently isolated, with failures popping back up as you work through them.

However specific does not necessarily mean uncommon. I think that you're right, without further evidence of this being wanted, there aren't enough perspectives to drive it towards a one-size-fits-all solution. But I couldn't find evidence of this feature being requested before to plus-one it, so now there's a place for it to accrue momentum and attract discussion if it's going to.

Until that happens, closing for now. Thanks for letting me bounce the idea off you!