Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

exit code 0 when there are exceptions prior to tests #41

Closed
nbogie opened this Issue May 4, 2011 · 4 comments

Comments

Projects
None yet
2 participants

nbogie commented May 4, 2011

This problem is demonstrated by putting the following code in spec/spec_helper.rb of a rails project.

require 'simplecov'
SimpleCov.start 'rails'
raise "some exception in the class loading before the tests start"

When rspec is invoked via rake, the above exception is logged to stderr and no tests are run, but the rake process exits 0 (i.e. success), where it should exit non-zero (failure) as it does when the SimpleCov.start is omitted.

Context:
This is a simplified version of the problem we encountered. We had a CI box that didn't receive a new config file required by a codebase change. When rake (rspec) ran, our app initialization code correctly raised an exception to protest but because our test suite was running under simplecov the rake (rspec) task exited 0 (success) indicating a passing build.

Environment:
ruby 1.9.2-p180,
simplecov gem 0.4.2
rspec gem 2.5.0
rubygems 1.6.2

Extra information:
I note that if I register an at_exit handler which does NOT call SimpleCov.result.format!, the exit code is propogated correctly (though of course no report will be produced). E.g. SimpleCov.at_exit { puts "simplecov at_exit handler - doing nothing" }

Thanks for any help, and for your work on this gem!

Owner

colszowka commented May 4, 2011

Thanks for the report. I'll have a look into it, though I can't come up with an easy solution right now.

nbogie commented May 4, 2011

Hi, I've made a fix for this issue but it currently lacks automated tests:
nbogie/simplecov@ec8bd54

In the manual test documented in the above issue, simplecov's at_exit() will be reached with $! populated with some exception other than SystemExit (specifically a RuntimeError). Simplecov was only calling exit explicitly if it found a SystemExit exception was present. I extended that to work for other exceptions. For some reason, the presence of $! was not anyway causing a non-zero exit, at least in the situation where the registered at_exit handler had to prepare a simplecov report. I haven't found good documentation on this aspect of ruby (pointers welcome).

I'm dubious of the existing approach of doing the reporting work (and of calling exit) from within an at_exit handler. Also, what happens if the report formatter raises an exception, given that it is also generally launched from this handler? Isn't there some other point at which to prepare the report rather than using the at_exit hook? I get that such a callback would likely be framework-dependent.

I've only glanced at the rest of the code; please excuse my ignorance.

Resources:

@nbogie nbogie closed this May 4, 2011

@nbogie nbogie reopened this May 4, 2011

Owner

colszowka commented Jun 3, 2011

I don't think there is a better way to capture the coverage results than with an at_exit handler - What other way could there be to let simplecov do it's thing when the test suite is finished?

Your commit makes a lot of sense to me, keeping this open so I remember to add tests and pull it in.

@nightscape nightscape pushed a commit to crealytics/simplecov that referenced this issue Sep 27, 2011

Martin Mauch Add test and fix for github issue #41 5ece56c

@colszowka colszowka added a commit that referenced this issue Sep 28, 2011

@colszowka colszowka Merge pull request #84 from crealytics/master
Fix issue #41
24ced99
Owner

colszowka commented Oct 12, 2011

This is included in the new 0.5.4 release, thanks a lot!

@colszowka colszowka closed this Oct 12, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment