Background and Problem
RSpec has a feature that I find very useful which is the
--only-failures option. This will re-run only that examples that failed the previous run.
CircleCI has support for uploading artifcats with your builds, which allows us to store the persistance file that powers the RSpec only failures option.
However! CircleCI also supports and encourages parallelizing your build, which means even if you upload your rspec persistance file, you actually have a number of them each containing a subset of your test suite.
This is where
Sleet comes in!
This tool does two things:
- It downloads all of the
.rspec_failed_examplesfiles that were uploaded to CircleCI for the most recent build of the current branch
- It combines the multiple files into a single sorted
.rspec_failed_examplesfile, and moves it to the current directory
1. Configure RSpec to Create and Use an example persistance file
We need to set the
example_status_persistence_file_path config in RSpec. Here are the relevant RSpec docs.
The first step is to create(/or add to) your
spec/spec_helper.rb file. We want to include the following configuration, which tells RSpec where to store the status persistance file. The actual location and file name are up to you, this is just an example. (Though using this name will require less configuration later.)
RSpec.configure do |c| c.example_status_persistence_file_path = ".rspec_example_statuses" end
if you just created the
spec_helper.rb file then you will need to create a
.rspec file containing the following to load your new helper file.
Again there are other ways to load your
spec_helper.rb file, including requiring it from each spec. Pick one that works for you.
2. Collect the example persistance files in CircleCI
To do this we need to create a step which saves the
.rspec_example_statuses as artifacts of the build. The following is an example of such a step in CircleCI. This must happen after rspec has run or else the persistance file will not exist.
- store_artifacts: path: .rspec_example_statuses
3. Save a CircleCI Token locally (to access private builds)
In order to see private builds/repos in CircleCI you will need to get a CircleCI token and save it locally to a Sleet Configuration file.
The recommended approach is to create a yml file in your home directory which contains your the key
An API token can be generated here: https://circleci.com/account/api
4. Run this tool from your project
This will look up the latest completed build in CircleCI for this branch, and download all the relevant
.rspec_example_statuses files. It then combines and sorts them and saves the result to the
.rspec_example_statuses file locally.
5. Run RSpec with
bundle exec rspec --only-failures
This will run only the examples that failed in CircleCI!
If you are using Worklfows in your CircleCI builds, or you are working with a different persistance file name, you may need to configure Sleet beyond the defaults.
Sleet currently supports two ways to input configurations:
- Through YML files
Sleetwill search 'up' from where the command was run and look for
.sleet.ymlfiles. It will combine all the files it finds, such that 'deeper' files take presedence. This allows you to have a user-level config at
~/.sleet.ymland have project specific files which take presendence over the user level config (ex:
- Through the CLI
- These always take presendece the options provided in the YML files
To view your current configuration use the
sleet config command which will give you a table of the current configuration. You can also use the
--print-config flag with the
fetch command to print out the config, including any other CLI options. This can be useful for bebugging as the output also tells you where each option came from.
These are the options that are currently supported
This is the directory of the source git repo. If a
source_dir is NOT given we look up from the current directory for the nearest git repo.
This is the name of the Rspec Circle Persistance File in CircleCI. The default is
This will match if the full path on CircleCI ends in the given name.
This is the name for the output file, on your local system. It is relative to the
Will be IGNORED if
workflows is provided.
If you are using workflows in CircleCI, then this is for you! You need to tell
Sleet which build(s) to look in, and where each output should be saved.
The input is a hash, where the key is the build name and the value is the
output_file for that build. Sleet supports saving the artifacts to multiple builds, meaning it can support a mono-repo setup.
For this example you have three jobs in your CircleCI Workflow,
deploy, but only 1 (the
test build) generate an Rspec persistance file
This command will pick the
test build and save it's artificats to the
sleet fetch --workflows test:.rspec_example_statuses
If you have a mono-repo that contains 3 sub-dirs.
baz. And each one has an accompanying build. We can process all these sub-dirs at once with the following workflow command.
sleet fetch --workflows foo-test:foo/.rpsec_example_statuses bar-test:bar/.rspec_example_statuses baz-specs:baz/spec/examples.txt