Skip to content

Unexpected coverage calculation after adding specs #1120

Open
@tgentry-bamboo

Description

@tgentry-bamboo

Hello, I'm working on a fairly large Rails application utilizing SimpleCov for test coverage reporting.

We are using SimpleCov version 0.22 and Rails 6.1.7.7.

Here is the SimpleCov configuration we are using as seen in our config/spec_helper.rb file:

SimpleCov.start :rails do
    add_filter ['/vendor/', '/lib/pmp_credentials/']
    command_name  "RSpec #{rand(1000000)}"
    merge_timeout 3600   # increase default because suite takes long time
  end

We are also using a .simplecov configuration file whose contents are such:

class LineFilter < SimpleCov::Filter
  def matches?(source_file)
    source_file.lines.count < filter_argument
  end
end

SimpleCov.start :rails do
  use_merging true
  merge_timeout 3600
  add_filter '/.gem/'
  add_filter '/vendor/'
  add_filter '/config/'
  add_filter '/lib/tasks/'
  add_filter '/script/'
  add_filter '/lib/*.rb'
  add_group 'Services', 'app/services'
  add_group 'Workers', 'app/workers'
  add_group 'Trailblazer', 'app/concepts'
  add_group 'Long files' do |src_file|
    src_file.lines.count > 100
  end
  add_group "Short files", LineFilter.new(5) # Using the LineFilter class defined in Filters section above
end

SimpleCov.minimum_coverage 80 if ENV['CHECK_COVERAGE']

The problem we are experiencing is that after adding ~100 new specs and making additions to an existing FactoryBot factory, using ~1000 new lines of spec code to do so, the coverage calculation has changed dropping us below our minimum coverage enforcement by 10%.

Here are the SImpleCov calculations before adding the new specs:

All Files ( 80.34% covered at 2594.63 hits/line )
1803 files in total.
52115 relevant lines, 41867 lines covered and 10248 lines missed. ( 80.34% )

and here are the SimpleCov calculations after adding the new specs:

All Files ( 71.91% covered at 3880.83 hits/line )
1803 files in total.
53406 relevant lines, 38406 lines covered and 15000 lines missed. ( 71.91% )

No new app code was added in this time, only the spec code mentioned previously. The amount of files has not changed but SimpleCov is picking up 1291 new lines of code and 3461 less lines covered with the simple addition of test code. This for us is unexpected behavior. I've searched for similar issues but was unable to find any. Any ideas off the top of the head for what the underlying issue may be? I'm willing to devote some time digging into SimpleCov's source code should the issue be an actual bug in the library but will need some guidance.

Thanks for all your hard work!!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions