parallel test runner for CI environments
Ruby Shell Gherkin
Latest commit 4a715e6 May 11, 2017 @tmm1 tmm1 Merge pull request #73 from w1mvy/fix_rspec_deprecated
Fix deprecated method at rspec3
Permalink
Failed to load latest commit information.
bin Require test files in workers instead of master Sep 16, 2016
lib Fix deprecated `:example_group` hash, when rspec3. use hash directry Mar 21, 2017
script Prune suites not seen in 8 days from .test_queue_stats Sep 15, 2016
spec Require test files in workers instead of master Sep 16, 2016
test Add a test for TEST_QUEUE_REMOTE_MASTER_MESSAGE Mar 9, 2017
.gitignore Fix cucumber support Aug 24, 2015
.travis.yml Prune suites not seen in 8 days from .test_queue_stats Sep 15, 2016
Gemfile Skip individual tests based on whether gems are installed Sep 8, 2016
Gemfile-cucumber1-3 Run tests for Cucumber 2.4 Sep 8, 2016
Gemfile-cucumber1-3.lock v0.4.2 Jan 17, 2017
Gemfile-cucumber2-4 Run tests for Cucumber 2.4 Sep 8, 2016
Gemfile-cucumber2-4.lock v0.4.2 Jan 17, 2017
Gemfile-minitest4 add tests for minitest5 Jun 24, 2014
Gemfile-minitest4.lock v0.4.2 Jan 17, 2017
Gemfile-minitest5 Test each test framework separately Sep 8, 2016
Gemfile-minitest5.lock v0.4.2 Jan 17, 2017
Gemfile-rspec2-1 Test each test framework separately Sep 8, 2016
Gemfile-rspec2-1.lock v0.4.2 Jan 17, 2017
Gemfile-rspec3-0 Gemfiles for different versions of RSpec May 11, 2015
Gemfile-rspec3-0.lock v0.4.2 Jan 17, 2017
Gemfile-rspec3-1 Gemfiles for different versions of RSpec May 11, 2015
Gemfile-rspec3-1.lock v0.4.2 Jan 17, 2017
Gemfile-rspec3-2 Gemfiles for different versions of RSpec May 11, 2015
Gemfile-rspec3-2.lock v0.4.2 Jan 17, 2017
Gemfile-testunit wip test-unit support Dec 29, 2014
Gemfile-testunit.lock v0.4.2 Jan 17, 2017
Gemfile.lock v0.4.2 Jan 17, 2017
README.md TEST_QUEUE_SLAVE_MESSAGE -> TEST_QUEUE_REMOTE_MASTER_MESSAGE Mar 7, 2017
test-queue.gemspec v0.4.2 Jan 17, 2017

README.md

test-queue

Gem Version Build Status

Yet another parallel test runner, built using a centralized queue to ensure optimal distribution of tests between workers.

Specifically optimized for CI environments: build statistics from each run are stored locally and used to sort the queue at the beginning of the next run.

design

test-queue uses a simple master + pre-fork worker model. The master exposes a unix domain socket server which workers use to grab tests off the queue.

─┬─ 21232 minitest-queue master
 ├─── 21571 minitest-queue worker [3] - AuthenticationTest
 ├─── 21568 minitest-queue worker [2] - ApiTest
 ├─── 21565 minitest-queue worker [1] - UsersControllerTest
 └─── 21562 minitest-queue worker [0] - UserTest

test-queue also has a distributed mode, where additional masters can share the workload and relay results back to a central master.

environment variables

  • TEST_QUEUE_WORKERS: number of workers to use per master (default: all available cores)
  • TEST_QUEUE_VERBOSE: show results as they are available (default: 0)
  • TEST_QUEUE_SOCKET: unix socket path (or tcp address:port pair) used for communication (default: /tmp/test_queue_XXXXX.sock)
  • TEST_QUEUE_RELAY: relay results back to a central master, specified as tcp address:port
  • TEST_QUEUE_STATS: path to cache build stats in-build CI runs (default: .test_queue_stats)
  • TEST_QUEUE_FORCE: comma separated list of suites to run
  • TEST_QUEUE_RELAY_TIMEOUT: when using distributed builds, the amount of time a remote master will try to reconnect to start work
  • TEST_QUEUE_RELAY_TOKEN: when using distributed builds, this must be the same on remote masters and the central master for remote masters to be able to connect.
  • TEST_QUEUE_REMOTE_MASTER_MESSAGE: when using distributed builds, set this on a remote master and it will appear in that master's connection message on the central master.
  • TEST_QUEUE_SPLIT_GROUPS: split tests up by example rather than example group. Faster for tests with short setup time such as selenium. RSpec only. Add the :no_split tag to ExampleGroups you don't want split.

usage

test-queue bundles testunit-queue, minitest-queue and rspec-queue binaries which can be used directly:

$ minitest-queue $(find test/ -name \*_test.rb)
$ rspec-queue --format progress spec

But the underlying TestQueue::Runner::TestUnit, TestQueue::Runner::MiniTest and TestQueue::Runner::RSpec are built to be subclassed by your application. I recommend checking a new executable into your project using one of these superclasses.

$ vim script/test-queue
$ chmod +x script/test-queue
$ git add script/test-queue

Since test-queue uses fork(2) to spawn off workers, you must ensure each worker runs in an isolated environment. Use the after_fork hook with a custom runner to reset any global state.

#!/usr/bin/env ruby

class MyAppTestRunner < TestQueue::Runner::MiniTest
  def after_fork(num)
    # use separate mysql database (we assume it exists and has the right schema already)
    ActiveRecord::Base.configurations['test']['database'] << num.to_s
    ActiveRecord::Base.establish_connection(:test)

    # use separate redis database
    $redis.client.db = num
    $redis.client.reconnect
  end

  def prepare(concurrency)
    # create mysql databases exists with correct schema
    concurrency.times do |i|
      # ...
    end

    # If this is a remote master, tell the central master something about us
    @remote_master_message = "Output for remote master 123: http://myhost.com/build/123"
  end

  def around_filter(suite)
    $stats.timing("test.#{suite}.runtime") do
      yield
    end
  end
end

MyAppTestRunner.new.execute

distributed mode

To use distributed mode, the central master must listen on a tcp port. Additional masters can be booted in relay mode to connect to the central master. Remote masters must provide a TEST_QUEUE_RELAY_TOKEN to match the central master's.

$ TEST_QUEUE_RELAY_TOKEN=123 TEST_QUEUE_SOCKET=0.0.0.0:12345 bundle exec minitest-queue ./test/sample_test.rb
$ TEST_QUEUE_RELAY_TOKEN=123 TEST_QUEUE_RELAY=0.0.0.0:12345  bundle exec minitest-queue ./test/sample_test.rb
$ TEST_QUEUE_RELAY_TOKEN=123 ./test-multi.sh

See the Parameterized Trigger Plugin for a simple way to do this with jenkins.

see also