Skip to content

Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber

Notifications You must be signed in to change notification settings

kantox/parallel_tests

 
 

Repository files navigation

parallel_tests

Gem Version Build Status Build status

Speedup Test::Unit + RSpec + Cucumber + Spinach by running parallel on multiple CPU cores.
ParallelTests splits tests into even groups (by number of lines or runtime) and runs each group in a single process with its own database.

Setup for Rails

RailsCasts episode #413 Fast Tests

Install

Gemfile:

gem 'parallel_tests', group: [:development, :test]

Add to config/database.yml

ParallelTests uses 1 database per test-process.

Process number123
ENV['TEST_ENV_NUMBER']'''2''3'
test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Create additional database(s)

rake parallel:create

Copy development schema (repeat after migrations)

rake parallel:prepare

Setup environment from scratch (create db and loads schema, useful for CI)

rake parallel:setup

Run!

rake parallel:test          # Test::Unit
rake parallel:spec          # RSpec
rake parallel:features      # Cucumber
rake parallel:features-spinach       # Spinach

rake parallel:test[1] --> force 1 CPU --> 86 seconds
rake parallel:test    --> got 2 CPUs? --> 47 seconds
rake parallel:test    --> got 4 CPUs? --> 26 seconds
...

Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)

rake parallel:test[^test/unit] # every test file in test/unit folder
rake parallel:test[user]  # run users_controller + user_helper + user tests
rake parallel:test['user|product']  # run user and product related tests
rake parallel:spec['spec\/(?!features)'] # run RSpec tests except the tests in spec/features

Example output

2 processes for 210 specs, ~ 105 specs per process
... test output ...

843 examples, 0 failures, 1 pending

Took 29.925333 seconds

Run an arbitrary task in parallel

RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake parallel:rake[my:custom:task]
# limited parallelism
rake parallel:rake[my:custom:task,2]

Running things once

require "parallel_tests"

# preparation:
# affected by race-condition: first process may boot slower than the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)

# cleanup:
# last_process? does NOT mean last finished process, just last started
ParallelTests.last_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Even test group run-times

Test groups are often not balanced and will run for different times, making everything wait for the slowest group. Use these loggers to record test runtime and then use the recorded runtime to balance test groups more evenly.

RSpec

Rspec: Add to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

To use a custom logfile location (default: tmp/parallel_runtime_rspec.log), use the CLI: parallel_test spec -t rspec --runtime-log my.log

Minitest

Add to your test_helper.rb:

require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']

results will be logged to tmp/parallel_runtime_test.log when RECORD_RUNTIME is set, so it is not always required or overwritten.

Loggers

RSpec: SummaryLogger

Log the test output without the different processes overwriting each other.

Add the following to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log

RSpec: FailuresLogger

Produce pastable command-line snippets for each failed example. For example:

rspec /path/to/my_spec.rb:123 # should do something

Add to .rspec_parallel or use as CLI flag:

--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log

(Not needed to retry failures, for that pass --only-failures to rspec)

Cucumber: FailuresLogger

Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.

Usage:

cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Or add the formatter to the parallel: profile of your cucumber.yml:

parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Note if your cucumber.yml default profile uses <%= std_opts %> you may need to insert this as follows parallel: <%= std_opts %> --format progress...

To rerun failures:

cucumber @tmp/cucumber_failures.log

Setup for non-rails

gem install parallel_tests
# go to your project dir
parallel_test test/
parallel_rspec spec/
parallel_cucumber features/
parallel_spinach features/
  • use ENV['TEST_ENV_NUMBER'] inside your tests to select separate db/memcache/etc.

  • Only run selected files & folders:

    parallel_test test/bar test/baz/foo_text.rb

  • Pass test-options and files via --:

    parallel_test -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance

Options are:

-n [PROCESSES]                   How many processes to use, default: available CPUs
-p, --pattern [PATTERN]          run tests matching this regex pattern
    --exclude-pattern [PATTERN]  exclude tests matching this regex pattern
    --group-by [TYPE]            group tests by:
      found - order of finding files
      steps - number of cucumber/spinach steps
      scenarios - individual cucumber scenarios
      filesize - by size of the file
      runtime - info from runtime log
      default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, --single [PATTERN]           Run all matching files in the same process
-i, --isolate                    Do not run any other tests in the group used by --single(-s).
                                 Automatically turned on if --isolate-n is set above 0.
    --isolate-n                  Number of processes for isolated groups. Default to 1 when --isolate is on.
    --only-group INT[,INT]
-e, --exec [COMMAND]             execute this code parallel and with ENV['TEST_ENV_NUMBER']
-o, --test-options '[OPTIONS]'   execute test commands with those options
-t, --type [TYPE]                test(default) / rspec / cucumber / spinach
    --suffix [PATTERN]           override built in test file pattern (should match suffix):
      '_spec.rb$' - matches rspec files
      '_(test|spec).rb$' - matches test or spec files
    --serialize-stdout           Serialize stdout output, nothing will be written until everything is done
    --prefix-output-with-test-env-number
                                 Prefixes test env number to the output when not using --serialize-stdout
    --combine-stderr             Combine stderr into stdout, useful in conjunction with --serialize-stdout
    --non-parallel               execute same commands but do not in parallel, needs --exec
    --no-symlinks                Do not traverse symbolic links to find test files
    --ignore-tags [PATTERN]      When counting steps ignore scenarios with tags that match this pattern
    --nice                       execute test commands with low priority.
    --runtime-log [PATH]         Location of previously recorded test runtimes
    --allowed-missing [INT]      Allowed percentage of missing runtimes (default = 50)
    --unknown-runtime [FLOAT]    Use given number as unknown runtime (otherwise use average time)
    --first-is-1                 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
    --fail-fast                  Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
    --verbose                    Print debug output
    --verbose-process-command    Displays only the command that will be executed by each process
    --verbose-rerun-command      When there are failures, displays the command executed by each process that failed
    --quiet                      Print only tests output
-v, --version                    Show Version
-h, --help                       Show this.

You can run any kind of code in parallel with -e / --exec

parallel_test -n 5 -e 'ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]"'
hello from process "2"
hello from process ""
hello from process "3"
hello from process "5"
hello from process "4"
1 Process2 Processes4 Processes
RSpec spec-suite18s14s10s
Rails-ActionPack88s53s44s

TIPS

RSpec

Cucumber

  • Add a parallel: foo profile to your config/cucumber.yml and it will be used to run parallel tests
  • ReportBuilder can help with combining parallel test results
    • Supports Cucumber 2.0+ and is actively maintained
    • Combines many JSON files into a single file
    • Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.

General

Contribute your own gotchas to the Wiki or even better open a PR :)

Authors

inspired by pivotal labs

Michael Grosser
michael@grosser.it
License: MIT

About

Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Ruby 100.0%