Simplecov with minitest Unit tests?

[spatil]

I am trying to use simplecov with minitests unit testing, I configured my test_helper.rb

require 'simplecov'
SimpleCov.start 'rails' do
  add_filter '/spec/'
  add_filter '/config/'
  add_filter '/lib/'
  add_filter '/vendor/'

  add_group 'Controllers', 'app/controllers'
  add_group 'Models', 'app/models'
  add_group 'Helpers', 'app/helpers'
  add_group 'Mailers', 'app/mailers'
  add_group 'Views', 'app/views'
end

But when i run the test cases, it shows me all the methods from models/controllers are not covered, for which there are test cases I have written. Only before filters, relations are seen green rest is all red.

Do i need some configuration? is Simplecov is supported for Minitest unit testing ?

[spatil]

Sorry i think i noticed, If I run each file individually then it shows green

ruby -Itest test/controllers/admin/partners_controller_test.rb

But If run with

rake test

or

rake minitest:all

it does not work..

[thijsnado]

I was able to get it to work as follows:

namespace :test do
  task :coverage do
    require 'simplecov'
    SimpleCov.start 'rails' # feel free to pass block
    Rake::Task["test"].execute
  end
end

Did not work when I dropped require 'simplecov' in test_helper

[jirikolarik]

Thanks @moger777 it works :)

[colszowka]

This should work when the load order is correct (test_helper -> simplecov & start -> actual app code). I am not sure how minitest handles this by default though, so there may be something simplecov can do differently to make things work out of the box, at the very least meaning improving the docs. I believe what also should work if you use Bundler and it automatically requires your code would be to use the .simplecov config file. It will be automatically picked up when Bundler requires simplecov and should make the coverage report.

[sci-phi]

I had set up a new Rails 4 app (Ruby 2) with four models and three controllers with some basic tests. When I added SimpleCov to the test_helper as described it generated data for unit tests, but nothing for controllers, helpers,libraries or my custom resque definitions in app/jobs.

I also tried the change from the pull request #244 for addining minitest support, but it made no difference (seems it's just a text label?)

By moving it into a separate Rake task as described by @moger777 above, I was able to run the coverage task and generate a report that included everything.

[envygeeks]

This is definitely a load order issue in that the way Rails handles things with Minitest and the way RSpec sets up things with Rails is entirely different... rake simplecov produces proper tests and results, rake test -r './test/support/simplecov' produces proper results, however rake test where support/simplecov is loaded directly after setting up the RAILS_ENV and before I load anything else results in nothing being tracked.

Here is some other information:

[1] pry(main)> whereami

From: /home/jordon/development/envygeeks/hello/test/minitest/helper.rb @ line 3 :

      1: ENV["RAILS_ENV"] ||= "test"
      2: 
 => 3: binding.pry
      4: require_relative "../support/simplecov"
      5: require_relative "../../config/environment"
      6: require "factory_girl_rails"
      7: require "rails/test_help"
      8: require "minitest/luna"

[2] pry(main)> require File.expand_path("../config/environment", __FILE__) # => false

[1] pry(main)> whereami

From: /home/jordon/development/envygeeks/hello/test/minitest/helper.rb @ line 3 :

      1: # ENV["RAILS_ENV"] ||= "test"
      2: 
 => 3: binding.pry
      4: require_relative "../support/simplecov"
      5: require_relative "../../config/environment"
      6: require "factory_girl_rails"
      7: require "rails/test_help"
      8: require "minitest/luna"

[2] pry(main)> Rails.env # => "test"
[6] pry(main)> Hello # => Hello

[envygeeks]

Here is your workaround: https://gist.github.com/envygeeks/6629308

[ptyagi16]

Thanks @envygeeks! works
+1

[brianp]

I'm using MiniSpec with Guard and have the same problem. Running stand alone generates just fine but running through guard shows all the method bodies being skipped. The railties fix above did not work for me.
The Simplecov generation message gets printed to the console before the tests run:

13:56:55 - INFO - Guard::Minitest 1.1.0 is running, with Minitest::Unit 4.7.5!
13:56:55 - INFO - Running all tests
Coverage report generated for Unit Tests to /Users/brianp/Sites/myapp/coverage. 986 / 1718 LOC (57.39%) covered.
Loaded suite
Started
..........................................................................(continues)
Finished in 6.48651 seconds.

[szechyjs]

i get this same problem when running a fresh rails 4 application with the minitest-spec-rails gem. Simplecov shows up in the log first, before any test results.

[deivid-rodriguez]

Same thing here, seems to not work fine with either minitest-spec-rails or guard or both.

[mattiasottosson]

Same thing for me with using guard and/or minitest-spec-rails. @envygeeks, what was the workaround? Gist is removed 😢

[deivid-rodriguez]

@envygeeks it would be awesome to see your workaround!

[kopchickm]

I don't know about @envygeeks workaround, but I found something that worked for me (I'm using minitest-spec-rails), I found that removing :require => false from simplecov in the Gemfile fixed the issue.

[nTraum]

I can't get it to work with gem 'minitest-rails' either :(

Anyone aware of what exactly the issue is or how to work around this?

[zxiest]

@kopchickm 's solution does it for me. Thanks!

[khiemlam]

I am still having trouble with this. What is the solution or workaround? Thanks in advance!

[bf4]

This is a load order issue. SimpleCov must be required and started before any library code is loaded. This is a common problem when running tests via rake. See me comment at #314 (comment)

[envygeeks]

I think one should read the comments before doing silly shit like closing a ticket with a broad statement that applies but cannot feasibly be fixed by the user without simplecov doing some work to actually make the issue fixable by the user.

[bf4]

@envygeeks I apologize if I closed this erroneously. Would you mind summarizing what I'm missing? Still looks like a load order issue to me, but I'll happily be corrected and help out if you if wrong.

[envygeeks]

@bf4 it is a load order issue, there is no doubt about that, but it's a load order issue that was rather hard for the average user to fix because even if we required simplecov before everything else in our test helper (the same way we would with RSpec) it did not actually get required first, it landed after everything else which required me to inject it and hack it in with a railtie, so that it was always loaded when Rails loaded it's test crap. While it was easy to fix, one of my initial suggestions was for Simplecov to provide the Railtie and then people wouldn't even need to much more than require that in their app where Rails does it's own requires. require "simplecov/railtie".

As my comment above (way above maybe) stated, that no matter what I did in my helper it wouldn't work right, I either had to have rake require it long before it did anything else, or I had to use a Railtie.

[bf4]

Okay, so I think we're coming at this from different perspectives is all. I was saying that if you run your tests with rake, to ensure you require and start simplecov there, before any Rails code is required, rather than in a spec helper, whereas you're saying, since vanilla Rails apps require the app in the Rakefile, the 'convention over configuration' approach would be to work with Rails and provide a railtie.

A railtie sounds like a fine idea, though I, personally, think the better path is for the docs to be clearer on this matter, and to recommend using a .simplecov file. This was probably the sixth load-order issue I had looked at, so I agree it is a problem, at least in the docs.

I did click on your gist link above when first reading these issues and it didn't resolve, so perhaps that's part of our miscommunication, as well.

[jonmarinello]

The gist is missing now. Anyone know where it can be found?

[bf4]

@jonmarinello Do you have a bug or just curious? If a bug, please create a new issue. This issue is closed.

[jonmarinello]

Got it resolved. Thanks.

[elsurudo]

Right, so what is the workaround for this? I'm using 'minitest-rails-capybara', and coverage misses most (but not all) of my model files...

[bf4]

@elsurudo have you read the comments? simplecov must be started as early as possible. running via rake? require and start at the top before any app code is loaded. I'd recommend using a .simplecov file to ensure simplecov is started as soon as it is required.

[elsurudo]

Yes, I've read them. That's the thing. I'm including

require 'simplecov'
SimpleCov.start

right at the top of my test_helper.rb (I'm using "minitest-rails-capybara"), and still have the same issue.

The strange thing is it picks up some of my models, but not all of them...

[envygeeks]

I'm going to circle around to the issue I had with your statement but blew off because I didn't have time to care but this is the second time it's come up, how exactly is a .simplecov file going to solve load order issues with Rails and Minitest run through rake? I'm just as confused by the logic as I'm sure @elsurudo is.

[elsurudo]

And yes, exactly. From my understanding .simplecov is just a convenient place to put your config if you eg. use multiple test frameworks. It's just another step of indirection.

[envygeeks]

That said, what I did and still do for all my newer Rails is just create a Railtie and then require it early in my application.rb (before my actual application module.) Which is what I asked and suggested for Simplecov to do themselves to solve this issue.

module MyApp
  module Railties
    class SimpleCov < Rails::Railtie
      config.before_initialize do
        if Rails.env.test?
          require "simplecov"
          # If you need setup SimpleCov stuff here.
        end
      end
    end
  end
end

[bf4]

@envygeeks thanks for helping out here. I agree the docs needs to be better etc. but no one has fixed it yet.

So, first, background on how simplecov gets coverage, extracted from #314 (comment)

Where coverage comes from

SimpleCov uses the stdlib Coverage module

The Coverage module, when active, tracks when code is evaluated for the first time (i.e. when loaded or run )

SimpleCov starts and stops tracking coverage (via Coverage) when you run SimpleCov.start and SimpleCov.result, respectively.

Thus, when you run SimpleCov.start, load/require a file/files, then run SimpleCov.result (which by default runs via an exit hook when your code finishes running), you will see a certain amount of 'code coverage'. That is, simply loading or requiring a Ruby file will result in some 'code coverage'. If you were to load all files in your app, then run SimpleCov.start, then run your tests, you would only capture the additional coverage for code that is executed for the first time. Likewise, if you ran SimpleCov.start, then loaded all files in your app, then ran the tests, even non-tested files would show 'coverage' as they will have code executed simple by being loaded/required.

Where SimpleCov.start should happen

As describe above, as early as possible; it must be before any app code is run.

When you require 'simplecov', the .simplecov file, if present, will be executed.

Thus, if you, perhaps through Bundle.require require simplecov, but do not have a .simplecov file, coverage will not start until you explicitly run SimpleCov.start. In other words, by using a .simplecov file, you ensure that SimpleCov will start as soon as it is required.

Common Problems

Running Rails tests using rake often gives erroneous coverage because the app is loaded before SimpleCov.start is run in a test helper.

Rails and Railties

Vanilla Rails will evaluate all Railties before initializing the Rails app and its initializers. Thus, a Railtie is a reasonable way to hook into Rails initialization lifecycle to start SimpleCov before app code is run.

Right now, require 'simplecov/railtie' loads the file

module SimpleCov
  class Railtie < ::Rails::Railtie
    rake_tasks do
      load 'simplecov/railties/tasks.rake'
    end
  end
end

What should now be obvious is that this railtie does not require 'simplecov' nor does it call SimpleCov.start. @envygeeks is correct that missing the require is really a bug. (but checking the Rails env isn't necessary, because presumably you are only requiring simplecov in test or whenever you want to run it.)

require 'simplecov'
module SimpleCov
  class Railtie < ::Rails::Railtie
    rake_tasks do
      load 'simplecov/railties/tasks.rake'
    end
  end
end

and to get around where you put the SimpleCov.start in your code is why I consider using a .simplecov file a best practice.

Example .simplecov

# https://github.com/colszowka/simplecov#using-simplecov-for-centralized-config
# Maybe put some conditional here not to execute the code below unless ENV['COVERAGE'] == 'true'
SimpleCov.start do
  # see https://github.com/colszowka/simplecov/blob/master/lib/simplecov/defaults.rb
  load_profile 'rails' # load_adapter < 0.8
  coverage_dir 'tmp/coverage'
  # Use multiple 'command_names' to differentiate reports being merged in together
  command_name "rails_app_#{$$}" # $$ is the processid
  merge_timeout 3600 # 1 hour
  add_group "Jobs",        "app/jobs"
  add_group "Middleware",  "app/middleware"
  add_group "Serializers", "app/serializers"
  add_group "Engines",     "engines"
  add_group "Views",       "app/views"
  add_group "Long files" do |src_file|
    src_file.lines.count > 100
  end
  class MaxLinesFilter < SimpleCov::Filter
    def matches?(source_file)
      source_file.lines.count < filter_argument
    end
  end
  add_group "Short files", MaxLinesFilter.new(5)

  # Exclude these paths from analysis
  add_filter 'lib/plugins'
  add_filter 'vendor'
  add_filter 'bundle'
end

or even define your own profile that is backwards compatible

if SimpleCov.respond_to?(:profiles)
  SimpleCov.profiles
else
  SimpleCov.adapters
end.define 'my_app' do
  if defined?(load_profile)
    load_profile  'test_frameworks'
  else
    load_adapter 'test_frameworks'
  end
  coverage_dir 'tmp/coverage'
  # Use multiple 'command_names' to differentiate reports being merged in together
  command_name "rails_app_#{$$}" # $$ is the processid
  merge_timeout 3600 # 1 hour
end
if ENV['COVERAGE'] == 'true'
  SimpleCov.start 'my_app'
   if ENV['CODECLIMATE_REPO_TOKEN']
    begin
      require "codeclimate-test-reporter"
      # We run in parallel, so output results to disk via TO_FILE
      # and send at end, like tddium config
      # https://github.com/codeclimate/ruby-test-reporter/blob/master/lib/code_climate/test_reporter/formatter.rb#L18
      ENV['TO_FILE'] = 'true'
      SimpleCov.formatters = [
          SimpleCov::Formatter::HTMLFormatter,
          CodeClimate::TestReporter::Formatter
      ]
    rescue LoadError
      STDERR.puts "Could not loade CodeClimate SimpleCov Reporter"
    end
  end
end
SimpleCov.at_exit do
  File.open(File.join(SimpleCov.coverage_path, 'coverage_percent.txt'), 'w') do |f|
    f.write SimpleCov.result.covered_percent
  end
  SimpleCov.result.format!
end

This PR is for you:

If anyone wants to help put together a 'standard response' doc, that would be great. See e.g. the one nokogiri uses https://github.com/sparklemotion/nokogiri/blob/master/STANDARD_RESPONSES.md

[envygeeks]

Your assumptions neglect a very important fact: Where the Railtie comes from. You are right that "I" don't need the if if it comes from SimpleCov but it's presumably wrong if it comes from my own app and I do not like to attach if statements onto a require.

[elsurudo]

Thanks for the help and thorough responses, gentlemen. It's gotten me on the right track, and it is now just a matter of configuring things...

However, whenever I've used SimpleCov in the past, I dropped those two lines into my test helper, and I was good to go... I really think that the fact that you feel this issue requires a "standard response" is a problem in itself, since this is really the main use case of the gem. Shouldn't the 90% case warrant the simplest configuration possible?

[bf4]

Ok @envygeeks good point. I was assuming you meant that code in your app-railtie should exist in the simplecov one.

@elsurudo IMHO, the problem is in conflicting conventions. If someone is running tests via rake, then you want to start SimpleCov at the top of your Rakefile. If running via e.g. rspec, then you want to start at the top of your spec_helper. You can cover both scenarios by putting all your config in a .simplecov and just make sure you have a require 'simplecov' at the top of your Rakefile and/or your spec_helper.

I created issue #340 and updated some more info in there.. much of it should be in the README, not in a STANDARD RESPONSE, I think.

[envygeeks]

My mind tells me that the simplest solution is to upgrade the Railtie, have it require and detect a '.simplecov' file and load it or continue with a default Rails setup... then you tell people modify their Gemfile to :require => "simplecov/railtie" and then it solves Rakefile, rspec, rake test, rake spec and everything because the Railtie can require before the module creation (the code you actually care to test and the code that gets loaded before your code.)

It wouldn't be crazy to have SimpleCov detect .simplecov and if it doesn't find it then go ahead with a default SimpleCov.start since it can already assume it's Rails (because it is a Railtie afterall...) This simplifies everything entirely for the average user and requires nothing special to get started right away and if they ever need extra stuff there is a .simplecov file for that that overrides your default SimpleCov.start

[bf4]

@envygeeks no need, I don't think, for simplecov/railtie since require 'simplecov' does this at the bottom of the file, which I guess would be a problem if require 'rails' is done after require 'simplecov', which I think is why the idiom is usually to use load instead of require

# Load Rails integration (only for Rails 3, see #113)
require 'simplecov/railtie' if defined? Rails::Railtie

and the .simplecov detection is just part of the defaults being loaded. (I think I expanded on this a bit in #340)

[envygeeks]

The first part is fair enough but your latter statement does not solve or even address the core of my suggestions for making simplecov easier.

[bf4]

@envygeeks sorry, now I see what you're saying. Something like

  require 'simplecov'
  config.before_initialize do
    SimpleCov.start('rails') unless SimpleCov.running
  end

with maybe some guards or something around it.

[gamov]

I have missing model coverage if I place the SimpleCov.start in my test_helper.rb (with Minitest). I use config.before_initialize now.

[reitzig]

FWIW, in my simple gem it was relatively easy to get SimpleCov going.

1. Add s.add_development_dependency 'simplecov', '~> 0.16' to .gemspec.
2. bundle install
3. Create test/enable_coverage.rb with this content:

   require 'simplecov'

4. Create .simplecov with this content:

   SimpleCov.start do
   end

5. In Rakefile, edit the test task to include enable_coverage.rb at first position:

   desc 'Run tests'
   Rake::TestTask.new do |t|
     t.libs << 'my_gem'
     t.test_files = FileList['test/enable_coverage.rb', 'test/test*.rb']
     t.verbose = true
   end

6. Run rake test and happily find coverage reports in coverage/.

I guess further config would go into .simplecov, but that is the basics.
@bf4 Maybe add a very minimal example like this to the documentation? Scrolling through here, one might get the idea that the task at hand is all but impossible.

[PragTob]

There is already generous documentation in the README I feel.

Your step number 5 is framework specific. Also what is mentioned in 3. needs to be required somewhere.

PRs to improve the documentation are very welcome though :)

[reitzig]

@PragTob My issues with the README are as follows.

• I use a gemspec, not a Gemfile.
• I don't have a test helper. Rake just runs all the minitests.
• It doesn't mention .simplecov; the "hope SimpleCov.start is required first" approach seems fickle, and the long discussion above (among others) seems to agree.
• All the Rails references are irrelevant for Gem development, and potentially confusing.

Your step number 5 is framework specific

What do you mean? Which framework? This whole issue is about Minitest, if that's what you mean.

Also what is mentioned in 3. needs to be required somewhere.

Rake does that. Note that the file is mentioned in 5.

PRs to improve the documentation are very welcome though :)

Of course they are. At this point, I don't have a good overview over SimpleCov, though. I landed here googling how to get it working in my scenario, and I found the doc to be less than helpful; I had to cross-reference and handwave a lot of the commentary. Therefore, I left the above minimal example (with a reference on where to find it live and working) in the hope it might help the next googler.

[bf4]

Locked just because this is old and long and isn't progressing any further. I admit it's not really resolved in the way I'd like. Please open a new issue and reference this one if you'd like to restart the 'simplecov' and 'minitest' don't always play well together conversation (both run in at_exit blocks) , esp in Rails.