Skip to content
This repository

Automatically detect the usage of parallel tests and modify setup #64

Closed
colszowka opened this Issue August 06, 2011 · 13 comments

7 participants

Christoph Olszowka Tom Meier César Camacho Andrew Haines Dmytrii Nagirniak Altonymous Joseph Shraibman
Christoph Olszowka
Owner

See #61: For parallel_tests, it is neccessary to set the command name to something random like this SimpleCov.command_name "RSpec #{rand(100000)}" so parallel runs do not overwrite each other.

This is not obvious to users of the library and should be handled by SimpleCov automatically.

As a bonus, it would be great if instead of random numbers, some replicable ID would be available for each parallel test run, for example using a MD5 of the invoked shell command so that reported line hits do not add up along the way when the same test runs twice.

Tom Meier

Hi,

Where should I put this command_name in the specs for it to work? Immediately after the .start command in spec_helper?

It seems to be generating the coverage in between the split processes, and not merging.

Tom Meier

All good, added it to a before each as follows:

config.before(:each) do
    SimpleCov.command_name "RSpec:#{Process.pid.to_s}#{ENV['TEST_ENV_NUMBER']}"
end
Tom Meier
César Camacho

My original question was where this code block should be placed? I apologize for deleting it. Thanks for the super quick response tommeier!

config.before(:each) do
    SimpleCov.command_name "RSpec:#{Process.pid.to_s}#{ENV['TEST_ENV_NUMBER']}"
end
Tom Meier

I've actually gone with a less 'brick to the face' method for it, now i've split up tasks, such as unit tests, etc by rake tasks. Then I have a Jenkins CI box that parallelizes out the tasks individually. Adding this to the Rakefile:

Rake::Task.class_eval do
  #Required to ensure SimpleCov doesn't overwrite coverage shared values
  alias :execute_without_simplecov_command :execute
  def execute_and_set_simplecov_command args=nil
    ENV['COV_COMMAND'] = "rspec:#{name}" if ENV['COVERAGE']
    execute_without_simplecov_command(args)
  end
  alias :execute :execute_and_set_simplecov_command
end

So simplecov only groups and applies coverage based on the rake task groups, ie: where the custom split has been applied for rake tasks.

Jenkins CI can then merge all these together (from a shared dir). Currently finding it as a very accurate method to merge parallelised specs with SimpleCov.

Andrew Haines

For RSpec, I'm using

config.before(:each) { SimpleCov.command_name example.location }

to tag the coverage with a unique id per spec (namely the filename and line number, eg ./spec/models/something_spec.rb:123)

Dmytrii Nagirniak

How do you fail the build then? Here's what I do:

# spec/support/coverage.rb
if ENV['COV']
  puts "Running with test coverage"
  require 'simplecov'

  SimpleCov.start 'rails' do
    add_filter "lib/extensions"
    # bug: changing the coverage_dir here breaks coverage recording.
    at_exit do
      SimpleCov.result.format!
      threshold, actual = 97, SimpleCov.result.covered_percent
      if actual < threshold
        $stderr.puts "Coverage #{actual}% is below the threshold of #{threshold}%."
        exit 1
      end
    end
  end

  RSpec.configure do |config|
    config.before(:each) do
      SimpleCov.command_name "RSpec:#{Process.pid}#{ENV['TEST_ENV_NUMBER']}"
    end
  end
end

But unfortunately the results don't seem to be merged:


Coverage report generated for RSpec:711845, RSpec:711926, RSpec:711942, RSpec:711977 to proj/x/coverage. 2997 / 3817 LOC (78.52%) covered.
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:72:in `store_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:48:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:11:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:44:in `results'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:61:in `merged_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:49:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:10:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:44:in `results'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:61:in `merged_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:49:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:10:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'
Coverage report generated for RSpec:711845, RSpec:711904, RSpec:711926, RSpec:711942, RSpec:711977 to /Users/dnagir/proj/x/coverage. 3349 / 3817 LOC (87.74%) covered.
/Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse': 757: unexpected token at '' (MultiJson::DecodeError)
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/json-1.7.4/lib/json/common.rb:155:in `parse'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:20:in `resultset'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/result_merger.rb:72:in `store_result'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov.rb:48:in `result'
    from /Users/dnagir/proj/x/spec/support/coverage.rb:11:in `block (2 levels) in <top (required)>'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `call'
    from /Users/dnagir/.rvm/gems/ruby-1.9.3-p194/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in `block in <top (required)>'

I'm probably missing something, but not sure what exactly.

Altonymous

Because after running this a few times I started encountering issues with it building the coverage report. I figured out that deleting the coverage folder would get it working again. So for now I've added:

config.before(:suite) do
rm -rf #{Rails.root}/coverage
end

Joseph Shraibman

@dnagir you are seeing the same race condition that I'm seeing. Simplecov needs to lock the data file when it accesses it. I'm assuming in lib/simplecov/json.rb

Joseph Shraibman

Scratch that json.rb is the wrong place. I'm too tired to hunt it down right now.

Joseph Shraibman

It needs to be in lib/simplecov/result_merger.rb and maybe some other places.

Joseph Shraibman

PR #185 should totally fix this issue.

ronwsmith referenced this issue from a commit in centro/simplecov September 10, 2013
Merge in jshraibman-mdsol:master to fix issue #64 70d6c08
Christoph Olszowka
Owner

As @jshraibman-mdsol wrote an (embarassing) while ago, PR #185 should have fixed this problem. It will be released in a couple of minutes as part of version 0.8.0.pre2

Christoph Olszowka colszowka closed this September 11, 2013
Joerg Sonnenberger jsonn referenced this issue from a commit in jsonn/pkgsrc March 14, 2014
Update ruby-simplecov to 0.8.2.
v0.8.2, 2013-11-20 ([changes](colszowka/simplecov@v0.8.1...v0.8.2))
=====================

## Bugfixes

  * Replaced the locking behaviour [via lockfile gem](colszowka/simplecov#185) with
    plain Ruby explicit file locking when merging results. This should make simplecov merging to behave well
    on Windows again.
    See [#258](colszowka/simplecov#258) and
    [#223](colszowka/simplecov#223) (thanks to @tomykaira)

v0.8.1, 2013-11-10 ([changes](colszowka/simplecov@v0.8.0...v0.8.1))
=====================

## Bugfixes

  * Fixed a regression introduced in 0.8.0 - the Forwardable STDLIB module is now required explicitly.
    See [#256](colszowka/simplecov#256) (thanks to @kylev)

v0.8.0, 2013-11-10 ([changes](colszowka/simplecov@v0.7.1...v0.8.0))
=====================

**Note: Yanked the same day because of the regression that 0.8.1 fixes, see above**

## TL;DR

It's been way too long since the last official release 0.7.1, but this was partly due to it proving itself
quite stable in most circumstances. This release brings various further stability improvements to result set merging
(especially when working with parallel_tests), the configuration, source file encodings, and command name guessing.

The 0.8 line is the last one to cooperate with Ruby < 1.9. Starting with 0.9, SimpleCov will assume to be running in
Ruby 1.9+, and will not try to detect or bail silently on older Ruby versions. An appropriate deprecation warning
has been added.

## Features

  * Configuration blocks now have access to variables and methods outside of the block's scope.
    See [#238](colszowka/simplecov#238) (thanks to @ms-tg)
  * You can now have a global `~/.simplecov` configuration file.
    See [#195](colszowka/simplecov#195) (thanks to @spagalloco)
  * simplecov-html now uses the MIT-licensed colorbox plugin. Some adjustments when viewing source files,
    including retaining the currently open file on refresh have been added.
    See [simplecov-html #15](colszowka/simplecov-html#15) (thanks to @chetan)
  * Adds support for Rails 4 command guessing, removes default group `vendor/plugins`.
    See [#181](colszowka/simplecov#181) and
    [#203](colszowka/simplecov#203) (thanks to @semanticart and @phallstrom)
  * You can now load simplecov without the default settings by doing `require 'simplecov/no_defaults'`
    or setting `ENV['SIMPLECOV_NO_DEFAULTS']`. Check `simplecov/defaults` to see what preconfigurations are getting
    dropped by using this. See [#209](colszowka/simplecov#209) (thanks to @ileitch)
  * The result set merging now uses the `lockfile` gem to avoid race conditions.
    See [#185](colszowka/simplecov#185) (thanks to @jshraibman-mdsol).
  * Automatically detect the usage of parallel_tests and adjust the command name with the test env number accordingly,
    See [#64](colszowka/simplecov#64) and
    [#185](colszowka/simplecov#185) (thanks to @jshraibman-mdsol).

## Enhancements

  * Rename adapters to "profiles" given that they are bundles of settings. The old adapter methods are
    deprecated, but remain available for now.
    See [#207](colszowka/simplecov#207) (thanks to @mikerobe)
  * Tweaks to the automatic test suite naming. In particular, `rspec/features` should now
    be correctly attributed to RSpec, not Cucumber.
    See [#212](colszowka/simplecov#212) (thanks to @ersatzryan and @betelgeuse)
  * MiniTest should now be identified correctly by the command name guesser.
    See [#244](colszowka/simplecov#244) (thanks to @envygeeks)
  * Makes SimpleCov resilient to inclusion of mathn library.
    See [#175](colszowka/simplecov#175) and
    [#140](colszowka/simplecov#140) (thanks to @scotje)
  * Allow coverage_dir to be an absolute path.
  * See [#190](colszowka/simplecov#190) (thanks to @jshraibman-mdsol)
  * The internal cucumber test suite now uses Capybara 2.
    See [#206](colszowka/simplecov#206) (thanks to @infertux)
  * Work-arounds for the Coverage library shipped in JRuby 1.6 to behave in line with MRI.
    See [#174](colszowka/simplecov#174) (thanks to @grddev)
  * Fix warning: instance variable @exit_status not initialized.
    See [#242](colszowka/simplecov#242) and
    [#213](colszowka/simplecov#213) (thanks to @sferik and @infertux)

## Bugfixes

  * Correct result calculations for people using :nocov: tags.
    See [#215](colszowka/simplecov#215) (thanks to @aokolish)
  * Average hits per line for groups of files is now computed correctly.
    See [#192](http://github.com/colszowka/simplecov/pull/192) and
    [#179](http://github.com/colszowka/simplecov/issues/179) (thanks to @graysonwright)
  * Compatability with BINARY internal encoding.
    See [#194](colszowka/simplecov#194) and
    [#127](colszowka/simplecov#127) (thanks to @justfalter)
  * Special characters in `SimpleCov.root` are now correctly escaped before being used as a RegExp.
    See [#204](colszowka/simplecov#204) and
    [#237](colszowka/simplecov#237) (thanks to @rli9)
8e418e7
Joerg Sonnenberger jsonn referenced this issue from a commit in jsonn/pkgsrc March 14, 2014
Update ruby-simplecov to 0.8.2.
v0.8.2, 2013-11-20 ([changes](colszowka/simplecov@v0.8.1...v0.8.2))
=====================

## Bugfixes

  * Replaced the locking behaviour [via lockfile gem](colszowka/simplecov#185) with
    plain Ruby explicit file locking when merging results. This should make simplecov merging to behave well
    on Windows again.
    See [#258](colszowka/simplecov#258) and
    [#223](colszowka/simplecov#223) (thanks to @tomykaira)

v0.8.1, 2013-11-10 ([changes](colszowka/simplecov@v0.8.0...v0.8.1))
=====================

## Bugfixes

  * Fixed a regression introduced in 0.8.0 - the Forwardable STDLIB module is now required explicitly.
    See [#256](colszowka/simplecov#256) (thanks to @kylev)

v0.8.0, 2013-11-10 ([changes](colszowka/simplecov@v0.7.1...v0.8.0))
=====================

**Note: Yanked the same day because of the regression that 0.8.1 fixes, see above**

## TL;DR

It's been way too long since the last official release 0.7.1, but this was partly due to it proving itself
quite stable in most circumstances. This release brings various further stability improvements to result set merging
(especially when working with parallel_tests), the configuration, source file encodings, and command name guessing.

The 0.8 line is the last one to cooperate with Ruby < 1.9. Starting with 0.9, SimpleCov will assume to be running in
Ruby 1.9+, and will not try to detect or bail silently on older Ruby versions. An appropriate deprecation warning
has been added.

## Features

  * Configuration blocks now have access to variables and methods outside of the block's scope.
    See [#238](colszowka/simplecov#238) (thanks to @ms-tg)
  * You can now have a global `~/.simplecov` configuration file.
    See [#195](colszowka/simplecov#195) (thanks to @spagalloco)
  * simplecov-html now uses the MIT-licensed colorbox plugin. Some adjustments when viewing source files,
    including retaining the currently open file on refresh have been added.
    See [simplecov-html #15](colszowka/simplecov-html#15) (thanks to @chetan)
  * Adds support for Rails 4 command guessing, removes default group `vendor/plugins`.
    See [#181](colszowka/simplecov#181) and
    [#203](colszowka/simplecov#203) (thanks to @semanticart and @phallstrom)
  * You can now load simplecov without the default settings by doing `require 'simplecov/no_defaults'`
    or setting `ENV['SIMPLECOV_NO_DEFAULTS']`. Check `simplecov/defaults` to see what preconfigurations are getting
    dropped by using this. See [#209](colszowka/simplecov#209) (thanks to @ileitch)
  * The result set merging now uses the `lockfile` gem to avoid race conditions.
    See [#185](colszowka/simplecov#185) (thanks to @jshraibman-mdsol).
  * Automatically detect the usage of parallel_tests and adjust the command name with the test env number accordingly,
    See [#64](colszowka/simplecov#64) and
    [#185](colszowka/simplecov#185) (thanks to @jshraibman-mdsol).

## Enhancements

  * Rename adapters to "profiles" given that they are bundles of settings. The old adapter methods are
    deprecated, but remain available for now.
    See [#207](colszowka/simplecov#207) (thanks to @mikerobe)
  * Tweaks to the automatic test suite naming. In particular, `rspec/features` should now
    be correctly attributed to RSpec, not Cucumber.
    See [#212](colszowka/simplecov#212) (thanks to @ersatzryan and @betelgeuse)
  * MiniTest should now be identified correctly by the command name guesser.
    See [#244](colszowka/simplecov#244) (thanks to @envygeeks)
  * Makes SimpleCov resilient to inclusion of mathn library.
    See [#175](colszowka/simplecov#175) and
    [#140](colszowka/simplecov#140) (thanks to @scotje)
  * Allow coverage_dir to be an absolute path.
  * See [#190](colszowka/simplecov#190) (thanks to @jshraibman-mdsol)
  * The internal cucumber test suite now uses Capybara 2.
    See [#206](colszowka/simplecov#206) (thanks to @infertux)
  * Work-arounds for the Coverage library shipped in JRuby 1.6 to behave in line with MRI.
    See [#174](colszowka/simplecov#174) (thanks to @grddev)
  * Fix warning: instance variable @exit_status not initialized.
    See [#242](colszowka/simplecov#242) and
    [#213](colszowka/simplecov#213) (thanks to @sferik and @infertux)

## Bugfixes

  * Correct result calculations for people using :nocov: tags.
    See [#215](colszowka/simplecov#215) (thanks to @aokolish)
  * Average hits per line for groups of files is now computed correctly.
    See [#192](http://github.com/colszowka/simplecov/pull/192) and
    [#179](http://github.com/colszowka/simplecov/issues/179) (thanks to @graysonwright)
  * Compatability with BINARY internal encoding.
    See [#194](colszowka/simplecov#194) and
    [#127](colszowka/simplecov#127) (thanks to @justfalter)
  * Special characters in `SimpleCov.root` are now correctly escaped before being used as a RegExp.
    See [#204](colszowka/simplecov#204) and
    [#237](colszowka/simplecov#237) (thanks to @rli9)
5993ca7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.