Warn if any tests fail during baselining

[abingham]

In general we should expect all tests to pass on a baselining run. This is equivalent to expecting all tests to pass on un-mutated code! Look into the best way to warn the user in these cases.

Original idea by @Zitrax in #110.

[mbj]

I'd even go for failing if any test fails due this phase (by default). At least my tool for Ruby does go this way.

Its better for user education. Especially as baselining happens in parallel and most test suites by default cannot handle it at one or another point, its better to error out and force the user to think/fix that invariant.

This leads to less confusing end-user bug reports abut phantom coverage, and an overall better experience.

[atodorov]

+1 for failing everything if baseline fails.

Btw I think I'll start working on that ASAP b/c I've seen it in production. Check out the change at:
https://github.com/rhinstaller/pykickstart/pull/96/commits/6a34d59f094ff6ef0c27aec2fad28a38ba51e523L59

The unmodified version of the code is a typo (or copy&paste error). There are many similar __eq__ methods in the same project so I'm pretty confident in that. Once I've identified the missing test cases using Cosmic Ray (and also knowledge from testing the other __eq__ methods) I've updated my tests. Without the above change however running nose outside CR failed. I have expected to see the failure even with CR.

[floer32]

Just another 👍 with a reference here, PITest (Java) also fails when anything goes wrong in the baseline.