Always 100% test coverage with Ruby 2.0p353 and Rails 4.1beta #271

Closed
clupprich opened this Issue Dec 21, 2013 · 22 comments

Projects

None yet

9 participants

@clupprich

I'm running the latest official patch version of Ruby 2.0 (p353) together with the the revision e4b9f6c of simplecov on Rails 4.1beta. I've added simplecov to the top of spec_helper but when I run bin/rake spec, it always reports 100% test coverage and doesn't see files with no test coverage (like certain controllers, etc.). Anyone experiencing the same problems?

I tried adapting the Spork logic to Spring, but without luck.

@clupprich

Hm, just tried running without Spring (via rake spec) and it still reports 100% test coverage and doesn't see any controllers.

@andruby
andruby commented Jan 3, 2014

I seem to have the same problem with regular tests with rails 4.1.0.beta1 both with and without spring.
Simplecov loading in test_helper.rb:

require 'simplecov'
SimpleCov.start 'rails' do
  add_group "Serializers", "app/serializers"
end
@mscharley

I'm not using rails, but had a similar issue #277 and fixed it by rolling back to ruby 1.9.3. Try that if possible.

@pmontrasio

As a side note, no problems with 2.1.0 x86_64, rvm 1.25.14, bundler 1.5.2, rails 4.0.2 on Linux.
I added the gem one hour ago to the Gemfile of a project.

gem 'simplecov', :require => false, :group => :test

bundle install
bundle exec rake spec

and it works.

$ bundle list |grep rspec
  * rspec-core (2.14.7)
  * rspec-expectations (2.14.4)
  * rspec-mocks (2.14.4)
  * rspec-rails (2.14.1)
$ bundle list |grep simplecov
  * simplecov (0.8.2)
  * simplecov-html (0.8.0)

I'm not using spring.

@clupprich

True, Ruby 2.1 works without a problem. Okay, I'll take that ;)

@bf4
Collaborator
bf4 commented Jul 13, 2014

Spring and SimpleCov don't mix well. Your Rakefile may also be loading library code. Also see my comment in #314 (comment)

@bf4 bf4 closed this Jul 13, 2014
@bf4
Collaborator
bf4 commented Oct 24, 2014

@swrobel PR to add this to the README?

@swrobel
Contributor
swrobel commented Oct 24, 2014
@swrobel
Contributor
swrobel commented Dec 2, 2014

Upon further testing of my solution, I've found that using simplecov with spring causes lower-than-actual coverage to be reported. I'm still stuck without a good solution to make the two work well together.

If it helps anyone knowledgeable with digging into the problem, I seem to get lower coverage for models when running simplecov through spring, but not for controllers.

With Spring:

COVERAGE:  77.78% -- 231/297 lines in 15 files

+----------+--------------------------------------------+-------+--------+---------------------------------------------+
| coverage | file                                       | lines | missed | missing                                     |
+----------+--------------------------------------------+-------+--------+---------------------------------------------+
|   4.35%  | app/models/company.rb                      | 23    | 22     | 1-2, 4, 6-7, 9-14, 16, 20-21, 24-25, 28-... |
|   7.69%  | app/models/email.rb                        | 13    | 12     | 1-2, 4-5, 7-11, 14, 16, 18                  |
|  43.48%  | app/models/user.rb                         | 46    | 26     | 1-3, 7-8, 10, 12-13, 15-21, 23, 27, 29, ... |
|  90.00%  | app/controllers/api/v1/base_controller.rb  | 10    | 1      | 15                                          |
|  90.62%  | app/controllers/api/v1/leaks_controller.rb | 32    | 3      | 13, 25, 38                                  |
|  94.74%  | app/controllers/dashboard_controller.rb    | 19    | 1      | 18                                          |
|  98.28%  | app/controllers/users_controller.rb        | 58    | 1      | 62                                          |
+----------+--------------------------------------------+-------+--------+---------------------------------------------+

Without Spring:

COVERAGE:  96.30% -- 286/297 lines in 15 files

+----------+--------------------------------------------+-------+--------+--------------------+
| coverage | file                                       | lines | missed | missing            |
+----------+--------------------------------------------+-------+--------+--------------------+
|  78.26%  | app/models/company.rb                      | 23    | 5      | 21, 25, 29, 33, 37 |
|  90.00%  | app/controllers/api/v1/base_controller.rb  | 10    | 1      | 15                 |
|  90.62%  | app/controllers/api/v1/leaks_controller.rb | 32    | 3      | 13, 25, 38         |
|  94.74%  | app/controllers/dashboard_controller.rb    | 19    | 1      | 18                 |
|  98.28%  | app/controllers/users_controller.rb        | 58    | 1      | 62                 |
+----------+--------------------------------------------+-------+--------+--------------------+
@chrishough

@swrobel I noticed SimpleCov now runs on every spring based rspec run. Is there a way to disable it or only run it if an ENV is set?

@sunnyrjuneja

@swrobel I have this same problem.

@sunnyrjuneja

@swrobel I've mounted Grape into my rails project and it is also displaying below correct coverage. See issue #363. In the next week or two, I'll start reading through the Spring source code to see if there's any reason that Spring is not correctly reporting coverage for models and rack mounted apps.

@bf4
Collaborator
bf4 commented Jan 30, 2015

A preloader that forks may allow you to make the Coverage module
behave as if it could clear tracked code. You should look at the
coverage.c code

@sunnyrjuneja

@bf4 That's helpful to know. Thanks.

@bf4
Collaborator
bf4 commented Jan 30, 2015

@chrishough If you wrap SimpleCov.start in a conditional, it will conditionally run. A common one people use is if ENV["COVERAGE"] =~ /\Atrue\z/i

@sunnyrjuneja

You can also only execute simplecov outside of Spring like this:

unless defined?(Spring)
  SimpleCov.start 'rails' do
  # ...
  end
end
@bf4
Collaborator
bf4 commented Jan 31, 2015

@whatasunnyday Here's a protip for that pattern

SimpleCov.start 'rails' do
  # other config
end unless defined?(Spring)
@sunnyrjuneja

@bf4 I've never seen that before 👍 . Thanks!

@chrishough

@bf4 @whatasunnyday the only way I was able to get this to work correctly with spring was to create the file config/spring.rb with the following in it:

if ENV['COVERAGE']
  require 'simplecov'
  SimpleCov.start do
    coverage_dir 'tmp/coverage'
    add_filter '/.bundle/'
    add_filter '/spec/'
    add_filter '/config/'
    add_group 'Models', 'app/models'
    add_group 'Controllers', 'app/controllers'
    add_group 'Services', 'app/services'
    add_group 'Utilities', 'app/utilities'
    add_group 'Helpers', 'app/helpers'
    add_group 'Libraries', 'lib'
    add_group 'Mailers', 'app/mailers'
    add_group "Long Files" do |src_file|
      src_file.lines.count > 300
    end
    add_group 'Ignored Code' do |src_file|
      File.readlines(src_file.filename).grep(/:nocov:/).any?
    end
  end
  SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
    SimpleCov::Formatter::HTMLFormatter
  ]
  SimpleCov.minimum_coverage 95
end
@bf4
Collaborator
bf4 commented Feb 2, 2015

@chrishough tl;dr you used an ENV var :) BTW, I recommend you move the simplecov config to a .simpelcov file in your app root.

@Pikomu
Pikomu commented Apr 8, 2015

@swrobel, adding SimpleCov.command_name 'Rspec' fixed the lower coverage issue, the file would be :

if ENV['RAILS_ENV'] == 'test'
  require 'simplecov'
  SimpleCov.start 'rails'
  SimpleCov.command_name 'Rspec'
end

Simplecov tries to discover the command but the spring command makes it impossible.
See these : http://stackoverflow.com/questions/17255475/making-spring-gem-work-with-simplecov.
And these : https://github.com/colszowka/simplecov#test-suite-names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment