Different code percentage across environments with same config

[andrer0cha]

Hi, congrats for the great work here!
I'm trying to setup my CI in a Bitbucket repo and to set a minimum code coverage using Simplecov. My issue is that the coverage is different between environments (local machine vs CI run on remote) even with same configs.

Local machine file paths extracted from coverage.json

app/controllers/application_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/helpers/application_helper.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",
config/application.rb",
config/boot.rb",
config/environment.rb",
config/environments/test.rb",
config/initializers/assets.rb",
config/initializers/content_security_policy.rb",
config/initializers/devise.rb",
config/initializers/filter_parameter_logging.rb",
config/initializers/inflections.rb",
config/initializers/permissions_policy.rb",
config/initializers/rolify.rb",
config/routes.rb",
spec/controllers/leads_controller_spec.rb",
spec/controllers/users/omniauth_callbacks_controller_spec.rb",
spec/factories/leads.rb",
spec/factories/roles.rb",
spec/factories/users.rb",
spec/models/lead_spec.rb",
spec/models/role_spec.rb",
spec/models/user_spec.rb",
spec/support/database_cleaner.rb",
spec/support/helpers/auth_helper.rb",

Remote CI file paths (extracted from coverage.json):

app/channels/application_cable/channel.rb",
app/channels/application_cable/connection.rb",
app/controllers/application_controller.rb",
app/controllers/graphql_controller.rb",
app/controllers/home_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/pages/hello_world_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/controllers/users/sessions_controller.rb",
app/graphql/marteleiro_schema.rb",
app/graphql/mutations/base_mutation.rb",
app/graphql/resolvers/base_resolver.rb",
app/graphql/types/base_argument.rb",
app/graphql/types/base_connection.rb",
app/graphql/types/base_edge.rb",
app/graphql/types/base_enum.rb",
app/graphql/types/base_field.rb",
app/graphql/types/base_input_object.rb",
app/graphql/types/base_interface.rb",
app/graphql/types/base_object.rb",
app/graphql/types/base_scalar.rb",
app/graphql/types/base_union.rb",
app/graphql/types/mutation_type.rb",
app/graphql/types/node_type.rb",
app/graphql/types/query_type.rb",
app/helpers/application_helper.rb",
app/jobs/application_job.rb",
app/mailers/application_mailer.rb",
app/models/ability.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",
config/application.rb",
config/boot.rb",
config/environment.rb",
config/environments/test.rb",
config/initializers/assets.rb",
config/initializers/content_security_policy.rb",
config/initializers/devise.rb",
config/initializers/filter_parameter_logging.rb",
config/initializers/inflections.rb",
config/initializers/permissions_policy.rb",
config/initializers/rolify.rb",
config/routes.rb",
spec/controllers/leads_controller_spec.rb",
spec/controllers/users/omniauth_callbacks_controller_spec.rb",
spec/factories/leads.rb",
spec/factories/roles.rb",
spec/factories/users.rb",
spec/models/lead_spec.rb",
spec/models/role_spec.rb",
spec/models/user_spec.rb",
spec/support/database_cleaner.rb",
spec/support/helpers/auth_helper.rb",

The coverage results are:

• Local: 98.18%
• CI Remote: 92.62%

To fix it I tried adding some filters to exclude the following paths:

# /spec_helper.rb

require 'simplecov'
require 'simplecov-json'

module SimpleCov
  module Formatter
    class MergedFormatter
      def format(result)
        SimpleCov::Formatter::HTMLFormatter.new.format(result)
        SimpleCov::Formatter::JSONFormatter.new.format(result)
      end
    end
  end
end
SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
SimpleCov.start 'rails' do
  add_filter 'config/'
  add_filter 'tmp/'
  add_filter 'public/'
  add_filter 'lib/'
  add_filter 'vendor/'
  add_filter 'storage/'
  add_filter 'db/'
  add_filter 'bin/'
end

...

But after that, the result was weirder:

app/channels/application_cable/channel.rb",
app/channels/application_cable/connection.rb",
app/controllers/application_controller.rb",
app/controllers/graphql_controller.rb",
app/controllers/home_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/pages/hello_world_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/controllers/users/sessions_controller.rb",
app/graphql/marteleiro_schema.rb",
app/graphql/mutations/base_mutation.rb",
app/graphql/resolvers/base_resolver.rb",
app/graphql/types/base_argument.rb",
app/graphql/types/base_connection.rb",
app/graphql/types/base_edge.rb",
app/graphql/types/base_enum.rb",
app/graphql/types/base_field.rb",
app/graphql/types/base_input_object.rb",
app/graphql/types/base_interface.rb",
app/graphql/types/base_object.rb",
app/graphql/types/base_scalar.rb",
app/graphql/types/base_union.rb",
app/graphql/types/mutation_type.rb",
app/graphql/types/node_type.rb",
app/graphql/types/query_type.rb",
app/helpers/application_helper.rb",
app/jobs/application_job.rb",
app/mailers/application_mailer.rb",
app/models/ability.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",

With a coverage of 18.88%

Currently my SimpleCov config is set at the beginning of the spec_helper.rb file but I also tried using .simplecov and had the same issues.

Dependencies' versions:

ruby (3.3.0)
rails (7.2.1)
rspec (3.13.0)
simplecov (0.22.0)
simplecov_json_formatter (0.1.4)

[rikas]

I have the same issue, did you figure this out @andrer0cha ?

[andrer0cha]

No, sir @rikas
I just put it aside to check it later in the project since it's a project that I'm working solo.
If you find the solution, please share with us.

[Braiao]

Same problem here, did you manage to solve it? @andrer0cha