Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn if any tests fail during baselining #111

Closed
abingham opened this issue Nov 7, 2015 · 3 comments

Comments

@abingham
Copy link
Contributor

commented Nov 7, 2015

In general we should expect all tests to pass on a baselining run. This is equivalent to expecting all tests to pass on un-mutated code! Look into the best way to warn the user in these cases.

Original idea by @Zitrax in #110.

@mbj

This comment has been minimized.

Copy link

commented Nov 7, 2015

I'd even go for failing if any test fails due this phase (by default). At least my tool for Ruby does go this way.

Its better for user education. Especially as baselining happens in parallel and most test suites by default cannot handle it at one or another point, its better to error out and force the user to think/fix that invariant.

This leads to less confusing end-user bug reports abut phantom coverage, and an overall better experience.

@atodorov

This comment has been minimized.

Copy link
Contributor

commented Sep 13, 2016

+1 for failing everything if baseline fails.

Btw I think I'll start working on that ASAP b/c I've seen it in production. Check out the change at:
https://github.com/rhinstaller/pykickstart/pull/96/commits/6a34d59f094ff6ef0c27aec2fad28a38ba51e523L59

The unmodified version of the code is a typo (or copy&paste error). There are many similar __eq__ methods in the same project so I'm pretty confident in that. Once I've identified the missing test cases using Cosmic Ray (and also knowledge from testing the other __eq__ methods) I've updated my tests. Without the above change however running nose outside CR failed. I have expected to see the failure even with CR.

@hangtwenty

This comment has been minimized.

Copy link

commented Sep 24, 2016

Just another 👍 with a reference here, PITest (Java) also fails when anything goes wrong in the baseline.

atodorov added a commit to MrSenko/cosmic-ray that referenced this issue Nov 3, 2016

Fail if baseline fails. Fixes sixty-north#111
Cosmic Ray exits with status 2 if baseline test execution fails.
Also prints on the console anything that was returned by the test
runner.

atodorov added a commit to MrSenko/cosmic-ray that referenced this issue Nov 3, 2016

Fail if baseline fails. Fixes sixty-north#111 and sixty-north#169
Cosmic Ray exits with status 2 if baseline test execution fails.
Also prints on the console anything that was returned by the test
runner.

@abingham abingham closed this in db7b7c6 Nov 8, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.