Skip to content
Automatically finds test-ordering bugs in RSpec test suites
Ruby
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
bin
features
lib
spec
.gitignore
.rspec
.travis.yml
Gemfile
LICENSE.txt
README.md
Rakefile
rspec-search-and-destroy.gemspec

README.md

RSpec Search-and-Destroy (SAD) runs your RSpec suites to automatically find the root causes of test ordering bugs.

Background

Ideally, tests are independent of each other, but sometimes global state can leak out of one test and cause another test to fail. When you have many tests, these types of failures can be insidious to debug.

Usage

First, add the appropriate hooks in your spec_helper.rb:

require 'rspec-search-and-destroy'

RSpec.configure do |config|
  RSpecSearchAndDestroy.configure(config)
end

Then run the driver program:

# Search and destroy mode
rspec-sad

# If you have a particular ordering that creates issues
SPEC_OPTIONS="--seed 12345" rspec-sad

# If you have a complicated script
rspec-sad --rspec-command "/path/to/script"

Search and destroy mode

In the search phase, SAD will run your test suite until a test failure occurs, then switch to the destroy phase. In the destroy phase, the contributing tests are narrowed down until a single test is found that causes the failure.

Caveats

RSpec SAD requires that your test suite not have any flaky tests. Any intermittently failing tests will cause false positives or false negatives. In this case, the results will not provide any useful information.

How it works

During the search phase, your test suite will be run once to get the set of tests that could contribute to the test failure. The order of tests will be saved and passed off to the destroy phase.

During each step of the destroy phase, half of the remaining tests will be disabled. If the test continues to fail, these disabled tests can be ignored. If the test stops failing, then the currently enabled tests can be enabled.

Implementation notes

The driver process runs RSpec as a subprocess. The two processes communicate using environment variables and files. The driver program specifies a set of tests to run, and RSpec reports the results back to the driver.

The test ordering and filtering is done via a custom config.order_examples block, and the results are saved using a custom formatter.

Something went wrong with that request. Please try again.