Different coverage for different rubies

[deivid-rodriguez]

I'm experiencing this issue in my gem: https://github.com/deivid-rodriguez/byebug, where I'm getting dramatically different coverage results in travis for ruby 2.0.0 (around 45%) and ruby 2.1.2 and ruby-head (around 85%). The result for ruby 2.0.0 is definitely the wrong one, so this is not that big deal, cause it seems to work fine in the latest versions.

But I'm not sure whether there's something to be fixed in simplecov and this is a really easy to reproduce one, so I thought I'd rather report it.

Cheers!

[deivid-rodriguez]

The problem with this is that the percentage shown in my badge is misleading and random (it just decreased 40% after a trivial commit), because I guess it depends on which coverage report is sent last.

[bf4]

Can you include a failing test or test code we can run directly? Do you think this is a bug in MRI? SimpleCov is a pretty light wrapper over the Ruby Coverage module (see http://ruby-doc.org/stdlib-2.0/libdoc/coverage/rdoc/Coverage.html https://github.com/ruby/ruby/blob/trunk/sample/coverage.rb https://github.com/ruby/ruby/blob/trunk/test/coverage/test_coverage.rb https://github.com/colszowka/simplecov/blob/master/lib/simplecov.rb )

# tl;dr of simplecov
module SimpleCov
  def self.start(profile=nil, &block)
    Coverage.start
  end
  def self.result
    @result ||= SimpleCov::Result.new(Coverage.result)
  end
  require 'simplecov/configuration'
  SimpleCov.send :extend, SimpleCov::Configuration
  # https://github.com/colszowka/simplecov/blob/master/lib/simplecov/configuration.rb
  module Configuration
    def at_exit(&block)
      @at_exit ||= Proc.new { SimpleCov.result.format! }
    end
  end
end
# https://github.com/colszowka/simplecov/blob/master/lib/simplecov/defaults.rb
at_exit do
  SimpleCov.at_exit.call
end

[deivid-rodriguez]

Hi @bf4, thanks for the reply, Yeah, maybe this is a bug in MRI, not sure... I wanted to have a look at it a week ago or so, it thought it could be related to byebug manipulating SCRIPT_LINES__ which AFAIK is used by code coverage tools. But I removed every reference to it in byebug and the problem's still there.

To reproduce, just run byebug's test suite (bundle exec rake) using ruby 2.0.0 and ruby 2.1.3 and see the huge difference in the report.

[deivid-rodriguez]

@bf4 After some additional though I tend to agree with you that this is most likely a bug in MRI. I'm guessing it has been fixed in the latest versions but not backported, that's why it's only reproduced in 2.0.

I've workaround this by not tracking coverage in 2.0 so I don't get more wrong 40% coverage badges.

I'm going to go ahead and close this. If I ever revisit this, I'll either reopen this issue or (most likely) report, request backport or whatever at ruby-core.

[bf4]

If you wanna be really gungho try this after you require simplecov

require 'json'
SimpleCov.at_exit do
  filename = File.join(SimpleCov.root, "coverage/all_#{RUBY_VERSION.gsub('.', '_')}.json")
  File.write filename, JSON.pretty_generate(Coverage.result.sort)
end

And compare the two for different ruby versions

[deivid-rodriguez]

Thanks for the nice tip. If I ever get to this again, I'll definitely use it!