SimpleCov affecting exit code of bacon

[fabiopelosin]

SimpleCov causes bacon to exit with a zero exit code even in case of failures/errors. The issue affects CI as failures are simply not reported and can go undetected.

The issue is present at least since version 0.8.1, but something more subtle might be going on.

---

As code speaks louder than words consider the following example:

• The build succeeds even with a failure
• SimpleCov is downgraded without any other chage
• The downgrade fixes an artificial issue created on purpose locally (locally I'm using Ruby 2.0.0) but not this specific one on Travis (this might be an error on my part or interference from the coveralls gem)
• SimpleCov is removed
• The build fails correctly

[tvdeyen]

Same problem for Alchemy CMS: https://travis-ci.org/magiclabs/alchemy_cms/jobs/16516963#L403

When I remove SimpleCov from spec_helper then the failing test suite exits with 1.
With simplecov they fail with 0, what causes travis to mark the builds as success.

0.8.1 and 0.8.2 both have this issue.

Downgrading to 0.7.1 fixes the problem.

My Gemfile: https://github.com/magiclabs/alchemy_cms/blob/master/Gemfile
My gemspec: https://github.com/magiclabs/alchemy_cms/blob/master/alchemy_cms.gemspec
My spec helper: https://github.com/magiclabs/alchemy_cms/blob/master/spec/spec_helper.rb

[colszowka]

:( It assumes some of the recent changes to lib/simplecov/defaults.rb must have caused this. I'll try to look into it soon!

[tvdeyen]

Ok, it's getting interesting :)

With simplecov 0.7.1 it fails for Rails 4.1.0.beta1 environments.
Simplecov 0.7.1 and Rails 3.2 is fine.

[jigfox]

I have the same issue with cucumber features, I had failing features, but the build still succeeded because of Exit Code '0'

[jasonroelofs]

I have also run into this problem with version 0.8.x. Downgrading to 0.7.1 fixes it.

[jasonroelofs]

I went looking through some of the history of lib/simplecov/defaults.rb and found one potential candidate for this bug: 5143ee6#diff-99ecb32ffe1919f41b2741f74b68dfa8L82

[colszowka]

@jasonroelofs Thanks - I'll try and revert the exit status change

[colszowka]

Simply reverting that particular change did not help :( This issue is very likely the same as #281, and although that one came later, there is more activity over there, so I'll close this in favor of the other one. Please go on with the discussion there :)