Can't get parallelization to work.

[vincentpaca]

I'm trying to configure my rails app to build in parallel tests and my shippable.yml currently looks like this.

# Build Environment
build_environment: Ubuntu 12.04

# language setting
language: ruby

# version numbers, testing against two versions of node
rvm:
  - 2.1.5

addons:
  firefox: "28.0"

services:
  - selenium

parallelized_test: true

cache: true

# env tag
env:
  global:
    - DISPLAY=:99.0
    - SPEC_OPTS="spec/suite.rb --color --require ci/reporter/rake/rspec_loader --format CI::Reporter::RSpecFormatter"
    - CI_REPORTS=shippable/testresults COVERAGE_REPORTS=shippable/codecoverage
    - DB=mysql2://shippable:@localhost/test
    - RAILS_ENV=test
  matrix:
    - RUN="rake" TEST_GROUP=1of9
    - RUN="rake" TEST_GROUP=2of9
    - RUN="rake" TEST_GROUP=3of9
    - RUN="rake" TEST_GROUP=4of9
    - RUN="rake" TEST_GROUP=5of9
    - RUN="rake" TEST_GROUP=6of9
    - RUN="rake" TEST_GROUP=7of9
    - RUN="rake" TEST_GROUP=8of9
    - RUN="rake" TEST_GROUP=9of9

# install dependencies
before_install:
  - "mysql -e 'create database if not exists test;'"

before_script:
  - "rm -rf $CI_REPORTS && mkdir -p $CI_REPORTS"
  - "rm -rf $COVERAGE_REPORTS && mkdir -p $COVERAGE_REPORTS"
  - "/etc/init.d/xvfb start"
  - rake -f CI.rake ci:copy_example_configs
  - rake -f CI.rake ci:update_db_yml
  - rake db:schema:load

script:
  - $RUN

after_script:
  - "/etc/init.d/xvfb stop"

Rakefile:

...
require 'ci/reporter/rake/rspec'
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new(:spec => ["ci:setup:rspec"]) do |t|false
  t.rspec_opts = ENV['SPEC_OPTS']
end
task :default => [:spec]
...

I can't make it play nice with parallel tests. I got the code coverage reports to work, but the test splitting doesn't.

Any insight would be really helpful.

[manishas]

@vincentpaca we will take a look and get back to you. Quick question - is the problem that the tests do not get split across the minions at all, or is it more that the report isn't being shown correctly?

[vincentpaca]

Hey @manishas , it definitely runs 9 different minions but it runs all the specs on each minion. SimpleCov is working properly and I can get it to show. The test tab on the other hand doesn't work.

[manishas]

Have you taken a look at our sample - https://github.com/shippableSamples/sample-rubyonrails-parallel-matrix-rspec for this scenario? That runs a subset of tests on each minion and outputs results to the test tab. Please make sure you follow the instructions there.

Also tagging @CADBOT to help further.

[vincentpaca]

That is where I actually based my configuration too. I'll give it another go now to make sure that I just didn't miss a config or something. Thanks @manishas.

[CADBOT]

Hey @vincentpaca ! While your shippable.yml file is looking correct, the problem is in the Rakefile. The shippable.yml file sets the different env vars for each minion in your parallel build, but your Rakefile must then use this variable to select a subset of the possible tests; without this additional bit of logic, each of your minions you create in the matrix will run all your tests. I'll breakdown how it works in the sample project Manisha linked you to in our sample

# Ported from https://github.com/cloudcastle/cucumber_in_groups
require 'active_support/core_ext/array/grouping'

if ENV['TEST_GROUP']
  ENV['TEST_GROUP'] =~ /^(\d+)of(\d+)$/
  group = $1.to_i
  groups_no = $2.to_i
  specs = (Dir['./**/*_spec.rb']).sort.in_groups(groups_no, false)
  specs_to_run = specs[group - 1]
else
  specs_to_run = Dir['./**/*_spec.rb']
end

specs_to_run.each do |file|
  require file
end

The above is the test code trigged by our rakefile. What it's doing is based on what value the TEST_GROUP env var is set to in this minion (being 1of3 2of3 or 3of3 in our case) a different set of tests will be ran. The $1 and $2 global variables are set by this line

ENV['TEST_GROUP'] =~ /^(\d+)of(\d+)$/

In our case for the TEST_GROUP=1of3 minion group = 1 and group_no = 3.
Then, we break our specs up into groups of 3

specs = (Dir['./**/*_spec.rb']).sort.in_groups(groups_no, false)

And based on the current group number, select one of the 3 groups we made

specs_to_run = specs[group - 1]

Finally, we iterate through the group of tests we selected for our minion, and run each one

specs_to_run.each do |file|
  require file
end

Sorry if that was a little long winded, but I wanted to make sure you have all the information you need to unblock yourself! If you need more info, feel free to reach out to me at charlie@shippable.com . We can set up a google hangout, skype, or phone call, and work our way through it!

[vincentpaca]

Hey @CADBOT , thanks for clearing that up. That helped a lot, and I've got the parallel specs running now. The issue was just suite.rb not getting the correct path for the spec files. This ticket can now be closed.

[CADBOT]

Glad to hear it @vincentpaca ! Keep in touch with any further questions or needs!

[vincentpaca]

Thanks @CADBOT . Sent you an email too.

[manishas]

@vincentpaca since you're working with @CADBOT and the parallelization works, so I'm closing this issue.