New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spork doesn't seem to run SimpleCov correctly with RSpec and Rails #42

Closed
fj opened this Issue May 6, 2011 · 41 comments

Comments

Projects
None yet
@fj

fj commented May 6, 2011

tl;dr:

Summary of results:

summary of issuesSpork
not runningrunning
SimpleCovtop of Spork.preforkworks as expectedSimpleCov doesn't run
top of Spork.each_runalways 0 LOC, 0% coverageSimpleCov doesn't run

I'm on an existing Rails project using Mongoid, Devise, RSpec, and Spork (0.9.0.rc5). Our spec/spec_helper.rb looks like this:

require 'spork'

ENV["RAILS_ENV"] ||= 'test'

Spork.prefork do
  require 'simplecov'
  SimpleCov.start 'rails'

  # trap mongoid
  require 'rails/mongoid'
  Spork.trap_class_method(Rails::Mongoid, :load_models)

  # trap devise
  require 'rails/application'
  Spork.trap_method(Rails::Application, :reload_routes!)

  require File.expand_path("../../config/environment", __FILE__)
  require 'rspec/rails'
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

  DatabaseCleaner.strategy = :truncation

  RSpec.configure do |config|
    config.mock_with :rspec

    config.before(:each) do
      DatabaseCleaner.clean
    end
  end
end

Spork.each_run do
  # reload factories
  Factory.definition_file_paths = [
    File.join(Rails.root, 'spec', 'factories')
  ]
  Factory.find_definitions
end

This configuration will run SimpleCov when Spork is not running, and it produces the expected result. When Spork is running, no coverage message is shown.

If I move the require 'simplecov'; SimpleCov.start 'rails' lines to the beginning of the Spork.each_run block, as SimpleCov's readme suggests, then when Spork is not running, I get this:

Coverage report generated for RSpec to /home/johnf/projects/<elided>/coverage. 0 / 0 LOC (0.0%) covered.

If Spork is running, then instead I get no coverage message, as before.

@fj

This comment has been minimized.

Show comment
Hide comment
@fj

fj May 6, 2011

FWIW, I'm running spork with bundle exec spork rspec, rake with bundle exec rake, and rspec with bundle exec rspec --drb spec.

fj commented May 6, 2011

FWIW, I'm running spork with bundle exec spork rspec, rake with bundle exec rake, and rspec with bundle exec rspec --drb spec.

@colszowka

This comment has been minimized.

Show comment
Hide comment
@colszowka

colszowka Jun 3, 2011

Owner

I'm not using Spork myself, but from what I understand is that it always spins up a new process when the test suite should be run.

The Ruby Coverage library SimpleCov uses works by tracking every required file AFTER it has been loaded. Therefore, if SimpleCov is loaded AFTER the Rails app and all of it's source files, it won't be able to see them (I think this is what you're getting with the each_run solution).

So I imagine the solution would be to make Spork require 'simplecov'; SimpleCov.start before loading the app's environment every time, so the actual code files become visible to SimpleCov. But as I said, I'm not using Spork and I'm not sure how exactly it does it's magic. Maybe the authors of Spork could shed some light onto this?

Owner

colszowka commented Jun 3, 2011

I'm not using Spork myself, but from what I understand is that it always spins up a new process when the test suite should be run.

The Ruby Coverage library SimpleCov uses works by tracking every required file AFTER it has been loaded. Therefore, if SimpleCov is loaded AFTER the Rails app and all of it's source files, it won't be able to see them (I think this is what you're getting with the each_run solution).

So I imagine the solution would be to make Spork require 'simplecov'; SimpleCov.start before loading the app's environment every time, so the actual code files become visible to SimpleCov. But as I said, I'm not using Spork and I'm not sure how exactly it does it's magic. Maybe the authors of Spork could shed some light onto this?

@colszowka

This comment has been minimized.

Show comment
Hide comment
@colszowka

colszowka Jun 3, 2011

Owner

Quick update: I pinged @timcharper about this, hopefully he can enlighten us a bit :)

Owner

colszowka commented Jun 3, 2011

Quick update: I pinged @timcharper about this, hopefully he can enlighten us a bit :)

@JeanMertz

This comment has been minimized.

Show comment
Hide comment
@JeanMertz

JeanMertz Jul 17, 2011

I too am having this issue. Seems not many people run into this problem, as I can't find anything anywhere else.

JeanMertz commented Jul 17, 2011

I too am having this issue. Seems not many people run into this problem, as I can't find anything anywhere else.

@turadg

This comment has been minimized.

Show comment
Hide comment
@turadg

turadg Sep 21, 2011

@fj, I bet your problem is your test environment caches classes. See @Estheth's comment in #30.

fwiw, I had been using the hack below in each_run and setting cache_classes to true but that doesn't work:

  # HACK so we can cache-classes in test.rb
  Dir["#{Rails.root}/app/**/*.rb"].each { |f| load f }

  require 'simplecov'
  SimpleCov.start 'rails'

turadg commented Sep 21, 2011

@fj, I bet your problem is your test environment caches classes. See @Estheth's comment in #30.

fwiw, I had been using the hack below in each_run and setting cache_classes to true but that doesn't work:

  # HACK so we can cache-classes in test.rb
  Dir["#{Rails.root}/app/**/*.rb"].each { |f| load f }

  require 'simplecov'
  SimpleCov.start 'rails'
@Godisemo

This comment has been minimized.

Show comment
Hide comment
@Godisemo

Godisemo Nov 7, 2011

I have this issue, i put the two lines for simplecov at the top of the Spork.prefork block. When I use guard no coverage report is generated but when I run rake specit works. Would be nice if it worked with guard though.

Godisemo commented Nov 7, 2011

I have this issue, i put the two lines for simplecov at the top of the Spork.prefork block. When I use guard no coverage report is generated but when I run rake specit works. Would be nice if it worked with guard though.

@denmarkin

This comment has been minimized.

Show comment
Hide comment
@denmarkin

denmarkin Nov 11, 2011

+1. Same applies to configuration with cucumber. I bet it's spork related

denmarkin commented Nov 11, 2011

+1. Same applies to configuration with cucumber. I bet it's spork related

@aantix

This comment has been minimized.

Show comment
Hide comment
@aantix

aantix Nov 28, 2011

+1. Same config as @Godisemo along with the same issue.

aantix commented Nov 28, 2011

+1. Same config as @Godisemo along with the same issue.

@sevenseacat

This comment has been minimized.

Show comment
Hide comment
@sevenseacat

sevenseacat Dec 2, 2011

+1 Same here. I swear, it used to work....

sevenseacat commented Dec 2, 2011

+1 Same here. I swear, it used to work....

@christophermanning

This comment has been minimized.

Show comment
Hide comment
@christophermanning

christophermanning Feb 7, 2012

+1

simplecov (0.5.4)
guard-rspec (0.5.10)
guard-spork (0.4.1)
spork (0.9.0)

christophermanning commented Feb 7, 2012

+1

simplecov (0.5.4)
guard-rspec (0.5.10)
guard-spork (0.4.1)
spork (0.9.0)

@colszowka

This comment has been minimized.

Show comment
Hide comment
@colszowka

colszowka Feb 9, 2012

Owner

Guys, this obviously is related to #60 - the offender being Spork's requirement to set config.cache_classes = false in config/environments/test.rb. Please try to set it to config.cache_classes = true. As I do not use Spork I don't know what the implications are for spork when you do this, but without it SimpleCov will fail to operate correctly :(

Owner

colszowka commented Feb 9, 2012

Guys, this obviously is related to #60 - the offender being Spork's requirement to set config.cache_classes = false in config/environments/test.rb. Please try to set it to config.cache_classes = true. As I do not use Spork I don't know what the implications are for spork when you do this, but without it SimpleCov will fail to operate correctly :(

@christophermanning

This comment has been minimized.

Show comment
Hide comment
@christophermanning

christophermanning Feb 9, 2012

@colszowka I had config.cache_classes set to true and false and it still wouldn't generate the coverage directory/report after guard finishes running tests. Additionally setting config.cache_classes = false in config/environments/test.rb causes ActiveRecord::AssociationTypeMismatch exceptions when using FactoryGirl: http://stackoverflow.com/questions/5954690/strange-activerecordassociationtypemismatch

christophermanning commented Feb 9, 2012

@colszowka I had config.cache_classes set to true and false and it still wouldn't generate the coverage directory/report after guard finishes running tests. Additionally setting config.cache_classes = false in config/environments/test.rb causes ActiveRecord::AssociationTypeMismatch exceptions when using FactoryGirl: http://stackoverflow.com/questions/5954690/strange-activerecordassociationtypemismatch

@sunaku

This comment has been minimized.

Show comment
Hide comment
@sunaku

sunaku Feb 9, 2012

Contributor

Something must be wrong in your setups. I successfully use both config.cache_classes=false and FactoryGirl to get code coverage results under Tork (not Spork).

Contributor

sunaku commented Feb 9, 2012

Something must be wrong in your setups. I successfully use both config.cache_classes=false and FactoryGirl to get code coverage results under Tork (not Spork).

@sunaku

This comment has been minimized.

Show comment
Hide comment
@sunaku

sunaku Feb 9, 2012

Contributor

You guys should try using the Coverage library directly in your setups. If it works, then SimpleCov needs fixing. Else, you know what to do. :)

Contributor

sunaku commented Feb 9, 2012

You guys should try using the Coverage library directly in your setups. If it works, then SimpleCov needs fixing. Else, you know what to do. :)

@timcharper

This comment has been minimized.

Show comment
Hide comment
@timcharper

timcharper Feb 9, 2012

I'm afraid I don't have any experience with SimpleCov to be able to shed much light on this. But, I can say this:

You shouldn't need config.cache_classes = false, although that's the easiest way to get your models and such to reload automatically. Read these wiki pages for more info:

https://github.com/sporkrb/spork/wiki/Troubleshooting (specifically: Some changes to files don’t take effect until I restart Spork.)

https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu

All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.

If a SimpleCov devel wants to do some remove pair programming with me to crack this together, I'd be happy to schedule some time.

timcharper commented Feb 9, 2012

I'm afraid I don't have any experience with SimpleCov to be able to shed much light on this. But, I can say this:

You shouldn't need config.cache_classes = false, although that's the easiest way to get your models and such to reload automatically. Read these wiki pages for more info:

https://github.com/sporkrb/spork/wiki/Troubleshooting (specifically: Some changes to files don’t take effect until I restart Spork.)

https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu

All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.

If a SimpleCov devel wants to do some remove pair programming with me to crack this together, I'd be happy to schedule some time.

@sunaku

This comment has been minimized.

Show comment
Hide comment
@sunaku

sunaku Feb 9, 2012

Contributor

On Thu 09 Feb 2012 10:44:56 AM PST, Tim Harper wrote:

You shouldn't need config.cache_classes = false, although that's
the easiest way to get your models and such to reload automatically.

All you should need to do is delay the initialization of SimpleCov
until after the process forks, and SimpleCov should be none the wiser.

That is correct. You should call Coverage.start() after forking. The
coverage results should be collected and processed by an at_exit hook
that was registered once before forking. Here is a working example:

https://github.com/sunaku/tork/blob/coverage/lib/tork/config/coverage.rb#L6-10

Contributor

sunaku commented Feb 9, 2012

On Thu 09 Feb 2012 10:44:56 AM PST, Tim Harper wrote:

You shouldn't need config.cache_classes = false, although that's
the easiest way to get your models and such to reload automatically.

All you should need to do is delay the initialization of SimpleCov
until after the process forks, and SimpleCov should be none the wiser.

That is correct. You should call Coverage.start() after forking. The
coverage results should be collected and processed by an at_exit hook
that was registered once before forking. Here is a working example:

https://github.com/sunaku/tork/blob/coverage/lib/tork/config/coverage.rb#L6-10

@chetan

This comment has been minimized.

Show comment
Hide comment
@chetan

chetan Feb 13, 2012

FWIW, this is working for me now. I initially had some issues and after a bunch of mucking about it now works (even though my config is right where it was when I started).

The one minor change I made to my code was to skip my custom initializer which bootstrapped some of my custom libs. I now require the initializer manually in the each_run block. This is the complete contents of my test_helper.rb (I'm using minitest):

https://gist.github.com/1820849

I also disabled cache_classes as noted above, though I'm not sure that's absolutely necessary.

chetan commented Feb 13, 2012

FWIW, this is working for me now. I initially had some issues and after a bunch of mucking about it now works (even though my config is right where it was when I started).

The one minor change I made to my code was to skip my custom initializer which bootstrapped some of my custom libs. I now require the initializer manually in the each_run block. This is the complete contents of my test_helper.rb (I'm using minitest):

https://gist.github.com/1820849

I also disabled cache_classes as noted above, though I'm not sure that's absolutely necessary.

@ches

This comment has been minimized.

Show comment
Hide comment
@ches

ches Feb 17, 2012

So I'm not familiar with Tork, but @sunaku is the ordering of that Coverage.start() hook before app files are (re)loaded dependent simply on internal load order within Tork? It seems like we'd need something akin to prepend_each_run in Spork to get things like SimpleCov to run immediately after forking and before (for instance) spork-rails' each_run blocks kick off any reloading, etc., does that sound right @timcharper or am I oversimplifying?

@chetan It's not clear what you're doing in the files you require from your Gist -- if you're loading your app env in each_run every time, you aren't getting much out of using Spork, no?

ches commented Feb 17, 2012

So I'm not familiar with Tork, but @sunaku is the ordering of that Coverage.start() hook before app files are (re)loaded dependent simply on internal load order within Tork? It seems like we'd need something akin to prepend_each_run in Spork to get things like SimpleCov to run immediately after forking and before (for instance) spork-rails' each_run blocks kick off any reloading, etc., does that sound right @timcharper or am I oversimplifying?

@chetan It's not clear what you're doing in the files you require from your Gist -- if you're loading your app env in each_run every time, you aren't getting much out of using Spork, no?

@chetan

This comment has been minimized.

Show comment
Hide comment
@chetan

chetan Feb 17, 2012

@ches oops, I forgot to attach test_prefork. Basically I bootstrap the whole rails and test environment w/ the exception of all app/ lib/ and test/ classes. My test runtime has gone from about 10sec (mostly in Bundle.setup) to 1sec. Updated the gist:

https://gist.github.com/1820849

chetan commented Feb 17, 2012

@ches oops, I forgot to attach test_prefork. Basically I bootstrap the whole rails and test environment w/ the exception of all app/ lib/ and test/ classes. My test runtime has gone from about 10sec (mostly in Bundle.setup) to 1sec. Updated the gist:

https://gist.github.com/1820849

@bashcoder

This comment has been minimized.

Show comment
Hide comment
@bashcoder

bashcoder Feb 28, 2012

I'm having the same issue, following the same logic grid as in the first report. I tried toggling class caching in the test environment to no avail. The coverage report just doesn't run unless I manually run rake spec. Everything works fine without spork.

Versions:

simplecov 0.6.1
guard 1.0.0
spork 0.9.0
rails 3.2.0

spec_helper.rb https://gist.github.com/1934043

bashcoder commented Feb 28, 2012

I'm having the same issue, following the same logic grid as in the first report. I tried toggling class caching in the test environment to no avail. The coverage report just doesn't run unless I manually run rake spec. Everything works fine without spork.

Versions:

simplecov 0.6.1
guard 1.0.0
spork 0.9.0
rails 3.2.0

spec_helper.rb https://gist.github.com/1934043

@bjnord

This comment has been minimized.

Show comment
Hide comment
@bjnord

bjnord Mar 7, 2012

rails (3.2.2)
rspec-rails (2.8.1)
simplecov (0.6.1)
spork (0.9.0)

For me, SimpleCov works when Spork is shut down, but not when Spork is running. I tried all the suggestions above (and from other Google searches on the same problem) to no avail; in particular, setting caching to true or false makes no difference. In the end, I learned two things:

  1. With Spork running, "rake simplecov" will cause the Spork.each_run section to be run twice. Whether that's the cause, I don't know, but in any case, all LOC in app/*/.rb show up in "not covered" red.
  2. However (also with Spork running), "rspec --no-drb spec" gives me a correct SimpleCov report, and Spork.each_run is only executed once.

Don't know if this will help anyone, but at least I have a way to get a coverage report without shutting Spork down every time.

bjnord commented Mar 7, 2012

rails (3.2.2)
rspec-rails (2.8.1)
simplecov (0.6.1)
spork (0.9.0)

For me, SimpleCov works when Spork is shut down, but not when Spork is running. I tried all the suggestions above (and from other Google searches on the same problem) to no avail; in particular, setting caching to true or false makes no difference. In the end, I learned two things:

  1. With Spork running, "rake simplecov" will cause the Spork.each_run section to be run twice. Whether that's the cause, I don't know, but in any case, all LOC in app/*/.rb show up in "not covered" red.
  2. However (also with Spork running), "rspec --no-drb spec" gives me a correct SimpleCov report, and Spork.each_run is only executed once.

Don't know if this will help anyone, but at least I have a way to get a coverage report without shutting Spork down every time.

@fredwu

This comment has been minimized.

Show comment
Hide comment
@fredwu

fredwu Mar 11, 2012

This works:

Spork.prefork do
  unless ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end

Spork.each_run do
  if ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end

fredwu commented Mar 11, 2012

This works:

Spork.prefork do
  unless ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end

Spork.each_run do
  if ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end
@fro

This comment has been minimized.

Show comment
Hide comment
@fro

fro Mar 13, 2012

@fredwu nice, it's working for me now.

fro commented Mar 13, 2012

@fredwu nice, it's working for me now.

@fivetwentysix

This comment has been minimized.

Show comment
Hide comment
@fivetwentysix

fivetwentysix Mar 15, 2012

@fredwu Works for me too!

ruby - 1.9.3
rails - 3.2.2
spork - 1.0.0rc

fivetwentysix commented Mar 15, 2012

@fredwu Works for me too!

ruby - 1.9.3
rails - 3.2.2
spork - 1.0.0rc

@bjnord

This comment has been minimized.

Show comment
Hide comment
@bjnord

bjnord Mar 23, 2012

@fredwu Excellent! That was the piece I needed to unlock the puzzle. There were two other things I had to do:

  1. I had to add the ":require => false" option to "gem 'simplecov'" in my Gemfile (this means "rake simplecov" no longer works but I don't care).
  2. As noted by @chetan above, I had "SimpleCov.coverage_dir(...)" in a config/initializers/*.rb file, and I had to move that into spec/spec_helper.rb next to "SimpleCov.start" -- I think that was causing SimpleCov to get loaded too soon.

Now I get identical results for "rake spec" and "rspec spec", with and without Spork running. Thank you! (I still have "config.cache_classes = false" by the way.)

bjnord commented Mar 23, 2012

@fredwu Excellent! That was the piece I needed to unlock the puzzle. There were two other things I had to do:

  1. I had to add the ":require => false" option to "gem 'simplecov'" in my Gemfile (this means "rake simplecov" no longer works but I don't care).
  2. As noted by @chetan above, I had "SimpleCov.coverage_dir(...)" in a config/initializers/*.rb file, and I had to move that into spec/spec_helper.rb next to "SimpleCov.start" -- I think that was causing SimpleCov to get loaded too soon.

Now I get identical results for "rake spec" and "rspec spec", with and without Spork running. Thank you! (I still have "config.cache_classes = false" by the way.)

@prusswan

This comment has been minimized.

Show comment
Hide comment
@prusswan

prusswan Apr 2, 2012

I am getting irregular coverage results with rspec tests while running against spork, despite applying @fredwu 's fix and playing with stuff like:

  1. set config.cache_classes = false
  2. add the ":require => false" option to "gem 'simplecov'"

For now I am simply going to leave it alone since the trouble proved to be too much for any potential gain (for those interested, my setup: https://github.com/prusswan/hw3_rottenpotatoes/tree/hw4)

prusswan commented Apr 2, 2012

I am getting irregular coverage results with rspec tests while running against spork, despite applying @fredwu 's fix and playing with stuff like:

  1. set config.cache_classes = false
  2. add the ":require => false" option to "gem 'simplecov'"

For now I am simply going to leave it alone since the trouble proved to be too much for any potential gain (for those interested, my setup: https://github.com/prusswan/hw3_rottenpotatoes/tree/hw4)

@fredwu

This comment has been minimized.

Show comment
Hide comment
@fredwu

fredwu Apr 2, 2012

@prusswan Your repo's Gemfile doesn't have :require => false. Also, I just checked my test.rb environment, it's config.cache_classes = true.

fredwu commented Apr 2, 2012

@prusswan Your repo's Gemfile doesn't have :require => false. Also, I just checked my test.rb environment, it's config.cache_classes = true.

@prusswan

This comment has been minimized.

Show comment
Hide comment
@prusswan

prusswan Apr 2, 2012

@fredwu as I said I have already tried changing those parameters but they did not improve matters, the result with rspec is pretty erratic.

prusswan commented Apr 2, 2012

@fredwu as I said I have already tried changing those parameters but they did not improve matters, the result with rspec is pretty erratic.

@rickyrobinson

This comment has been minimized.

Show comment
Hide comment
@rickyrobinson

rickyrobinson Jul 21, 2012

@prusswan and others, the solutions suggested here got me part of the way. I think the problem is when some of the code you want coverage reports for is run for the first time during Rails initialization (for example, if an initializer in config/initializers does some config on models or custom code in your lib directory or whatever). SimpleCov won't pick that stuff up because it's loaded before SimpleCov is. A solution is to split loading and initialization between prefork and each_run by requiring application.rb instead of environment.rb, and then explicitly initializing your app in each_run. This solution works with config.cache_classes = true and if threadsafe mode has not been turned on. A detailed blog post is here: http://rickyrobinson.id.au/2012/07/20/when-spork-puts-a-fork-in-your-cucumber-and-a-spanner-in-your-specs.

rickyrobinson commented Jul 21, 2012

@prusswan and others, the solutions suggested here got me part of the way. I think the problem is when some of the code you want coverage reports for is run for the first time during Rails initialization (for example, if an initializer in config/initializers does some config on models or custom code in your lib directory or whatever). SimpleCov won't pick that stuff up because it's loaded before SimpleCov is. A solution is to split loading and initialization between prefork and each_run by requiring application.rb instead of environment.rb, and then explicitly initializing your app in each_run. This solution works with config.cache_classes = true and if threadsafe mode has not been turned on. A detailed blog post is here: http://rickyrobinson.id.au/2012/07/20/when-spork-puts-a-fork-in-your-cucumber-and-a-spanner-in-your-specs.

@colszowka

This comment has been minimized.

Show comment
Hide comment
@colszowka

colszowka Jul 21, 2012

Owner

I still can't tell much about spork, but on the Ruby Toolbox I've been using spin(+guard-spin) with great success recently. It basically only preloads your app's gems (so probably there's a little time delay compared to spork), but works without any config, and it works smoothly with simplecov so far (of course, only mentioning coverage for the latetst spec that's been run - you can avoid this by expanding merge_timeout and setting a pseudo-random command_name)

Owner

colszowka commented Jul 21, 2012

I still can't tell much about spork, but on the Ruby Toolbox I've been using spin(+guard-spin) with great success recently. It basically only preloads your app's gems (so probably there's a little time delay compared to spork), but works without any config, and it works smoothly with simplecov so far (of course, only mentioning coverage for the latetst spec that's been run - you can avoid this by expanding merge_timeout and setting a pseudo-random command_name)

@rickyrobinson

This comment has been minimized.

Show comment
Hide comment
@rickyrobinson

rickyrobinson Jul 21, 2012

@colszowka Hey, spin looks pretty cool. Thanks for the pointer. It runs my specs just fine after I take out all the spork-related stuff, but not sure if it'll run my cukes. I note that spin takes the same approach to splitting loading and initialization that my spork hack takes:

We require config/application because that file (typically) loads Rails and any Bundler deps, as well as loading the initialization code for the app, but it doesn't actually perform the initialization. That happens in config/environment.

In my experience that's the best we can do in terms of preloading. Rails and the gem dependencies rarely change and so don't need to be reloaded. But you can't initialize the application because any non-trivial app will involve it's models/controllers, etc. in its initialization, which you definitely don't want to preload.

http://jstorimer.github.com/spin/#section-9

Update: got spin working with cucumber and made a few other changes: http://rickyrobinson.id.au/2012/07/24/spork-to-spin

rickyrobinson commented Jul 21, 2012

@colszowka Hey, spin looks pretty cool. Thanks for the pointer. It runs my specs just fine after I take out all the spork-related stuff, but not sure if it'll run my cukes. I note that spin takes the same approach to splitting loading and initialization that my spork hack takes:

We require config/application because that file (typically) loads Rails and any Bundler deps, as well as loading the initialization code for the app, but it doesn't actually perform the initialization. That happens in config/environment.

In my experience that's the best we can do in terms of preloading. Rails and the gem dependencies rarely change and so don't need to be reloaded. But you can't initialize the application because any non-trivial app will involve it's models/controllers, etc. in its initialization, which you definitely don't want to preload.

http://jstorimer.github.com/spin/#section-9

Update: got spin working with cucumber and made a few other changes: http://rickyrobinson.id.au/2012/07/24/spork-to-spin

@leckylao

This comment has been minimized.

Show comment
Hide comment
@leckylao

leckylao Jul 31, 2012

Having the simplecov runs fine with spork. But the report generated is missing the helper files. But once I stopped spork and run all the specs, then the report includes helper files. Any idea? Anyone have the same issue or know how to fix it?

leckylao commented Jul 31, 2012

Having the simplecov runs fine with spork. But the report generated is missing the helper files. But once I stopped spork and run all the specs, then the report includes helper files. Any idea? Anyone have the same issue or know how to fix it?

@phstc

This comment has been minimized.

Show comment
Hide comment
@phstc

phstc Sep 1, 2012

@fredwu's workaround works for me only if I don't add an add_filter in simplecov.

SimpleCov.start "rails" do
  add_filter "vendor"
end

When I do it I need to configure config.cache_classes = false in test.rb, otherwise I get /coverage. 0 / 0 LOC (0.0%) covered..

I don't feel comfortable disabling cache_classes, is that the correct way to fix it?

phstc commented Sep 1, 2012

@fredwu's workaround works for me only if I don't add an add_filter in simplecov.

SimpleCov.start "rails" do
  add_filter "vendor"
end

When I do it I need to configure config.cache_classes = false in test.rb, otherwise I get /coverage. 0 / 0 LOC (0.0%) covered..

I don't feel comfortable disabling cache_classes, is that the correct way to fix it?

@nfm

This comment has been minimized.

Show comment
Hide comment
@nfm

nfm Feb 4, 2013

An additional observation: If you're using ActiveRecord observers, and activating them from your config/application.rb, the models they observe will still be skipped, even with @fredwu's fix.

The observed models being skipped seems to only be an issue when I run my specs through Spork.

nfm commented Feb 4, 2013

An additional observation: If you're using ActiveRecord observers, and activating them from your config/application.rb, the models they observe will still be skipped, even with @fredwu's fix.

The observed models being skipped seems to only be an issue when I run my specs through Spork.

@aminariana

This comment has been minimized.

Show comment
Hide comment
@aminariana

aminariana Feb 21, 2013

Works for me:

I followed the "this works" suggestion by fredwu. It didn't work with Spork at first. Then I additionally followed bjnord's suggestion to add :required => false to the simplecov line in the Gemfile. That combination seems to have solved the problem for me. Now coverage metrics are being generated BOTH with and without Spork.

I should note though, that in my test.rb, the following is still unchanged, and that seems to not cause a difference. It still works without modification:
config.cache_classes = true

Further update:
Some of my models weren't showing up when running Rspec under Spork with Guard iteratively, but their dependencies were showing up. After trying a few things, it turns out both of the following are needed for simplecov to deterministically figure out the dependencies on every partial run:

In test.rb do:
config.cache_classes = false

In your spec, do:
require 'spec_helper'
require '<model>'

aminariana commented Feb 21, 2013

Works for me:

I followed the "this works" suggestion by fredwu. It didn't work with Spork at first. Then I additionally followed bjnord's suggestion to add :required => false to the simplecov line in the Gemfile. That combination seems to have solved the problem for me. Now coverage metrics are being generated BOTH with and without Spork.

I should note though, that in my test.rb, the following is still unchanged, and that seems to not cause a difference. It still works without modification:
config.cache_classes = true

Further update:
Some of my models weren't showing up when running Rspec under Spork with Guard iteratively, but their dependencies were showing up. After trying a few things, it turns out both of the following are needed for simplecov to deterministically figure out the dependencies on every partial run:

In test.rb do:
config.cache_classes = false

In your spec, do:
require 'spec_helper'
require '<model>'

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 15, 2013

I had the same issue with some application_helper lines that were not evaluated as covered code when running guard+spork. When running rspec without guard, coverage was 100% for that helper.

I use a gem that is located under vendor/gems at the moment which is not watched by guard, I excluded it with a filter in the .simplecov file.

I removed the vendor-filter and added :spec_paths => ["spec", "vendor/gems/.../spec"] to the Guardfile, so the gem-tests are now run together with my rails tests. With this configuration, the coverage is correct with AND without guard/spork.

ghost commented Mar 15, 2013

I had the same issue with some application_helper lines that were not evaluated as covered code when running guard+spork. When running rspec without guard, coverage was 100% for that helper.

I use a gem that is located under vendor/gems at the moment which is not watched by guard, I excluded it with a filter in the .simplecov file.

I removed the vendor-filter and added :spec_paths => ["spec", "vendor/gems/.../spec"] to the Guardfile, so the gem-tests are now run together with my rails tests. With this configuration, the coverage is correct with AND without guard/spork.

@buccolo

This comment has been minimized.

Show comment
Hide comment
@buccolo

buccolo Mar 22, 2013

SimpleCov will only pickup if we pass the --no-drb flag, as in:
bundle exec rspec spec/ --no-drb

Though I'm not sure why...

buccolo commented Mar 22, 2013

SimpleCov will only pickup if we pass the --no-drb flag, as in:
bundle exec rspec spec/ --no-drb

Though I'm not sure why...

@yvanross

This comment has been minimized.

Show comment
Hide comment
@yvanross

yvanross Sep 5, 2013

Here is how I solved the problem

in environment/test.rb
config.cache_classes = false

in spec_hepler.rb

Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
add_filter '/config/'
add_filter '/lib/'
add_filter '/vendor/'

add_group 'Controllers', 'app/controllers'
add_group 'Models', 'app/models'
add_group 'Helpers', 'app/helpers'
add_group 'Mailers', 'app/mailers'
add_group 'Views', 'app/views'

end

yvanross commented Sep 5, 2013

Here is how I solved the problem

in environment/test.rb
config.cache_classes = false

in spec_hepler.rb

Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
add_filter '/config/'
add_filter '/lib/'
add_filter '/vendor/'

add_group 'Controllers', 'app/controllers'
add_group 'Models', 'app/models'
add_group 'Helpers', 'app/helpers'
add_group 'Mailers', 'app/mailers'
add_group 'Views', 'app/views'

end

@kyletolle

This comment has been minimized.

Show comment
Hide comment
@kyletolle

kyletolle Oct 31, 2013

I am using Sinatra, RSpec, Guard, Spork, and SimpleCov, and I had the same issue where code metrics weren't being generated properly while running with Guard.

Posting here, with a complete spec_helper.rb file, in case other people try to do something similar with Sinatra.

This was my initial spec_helper, and the tip above where you checked for the DRB env var didn't work for me. This is what my spec_helper.rb started with. You can see the requiring files in the prefork block, which is what was the issue.

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require_files

  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end

Spork.each_run do
end

I used some of the ideas above to change it to what follows, which generates the correct code coverage metrics:

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end
end

Spork.each_run do
  require 'simplecov'
  SimpleCov.start do
    add_filter '/spec/'
  end

  require_files

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end

kyletolle commented Oct 31, 2013

I am using Sinatra, RSpec, Guard, Spork, and SimpleCov, and I had the same issue where code metrics weren't being generated properly while running with Guard.

Posting here, with a complete spec_helper.rb file, in case other people try to do something similar with Sinatra.

This was my initial spec_helper, and the tip above where you checked for the DRB env var didn't work for me. This is what my spec_helper.rb started with. You can see the requiring files in the prefork block, which is what was the issue.

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require_files

  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end

Spork.each_run do
end

I used some of the ideas above to change it to what follows, which generates the correct code coverage metrics:

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end
end

Spork.each_run do
  require 'simplecov'
  SimpleCov.start do
    add_filter '/spec/'
  end

  require_files

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end
@bf4

This comment has been minimized.

Show comment
Hide comment
@bf4

bf4 May 17, 2015

Collaborator

ref: #340

Collaborator

bf4 commented May 17, 2015

ref: #340

Repository owner locked and limited conversation to collaborators May 17, 2015

@bf4

This comment has been minimized.

Show comment
Hide comment
@bf4

bf4 May 17, 2015

Collaborator

Closing as stale

Collaborator

bf4 commented May 17, 2015

Closing as stale

@bf4 bf4 closed this May 17, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.