Stop mutation process

[BlackGad]

Is it possible to extend stryker flow with additional pre conditions? For example if any or specific unit test failed after initial test run?

Currently there is only one condition (50% of tests are failed).

[rouke-broersma]

I don't think we need to make this unnecessarily complex within stryker. if you have very specific preconditions, make them preconditions in whatever automation you use. We already provide testcase filter support, and we could provide support for not allowing any failing unit tests as well (binary option, allow failing unit tests default true, but no option to specify percentage of failing unit tests). If you would like support for not allowing any failing unit tests please create a feature request.

[psfinaki]

Hey there, a bit late to the party but first of all thanks for implementing this, second just wanted to link this discussion on the same topic I started back in a day.

My opinion is that the implemented solution can probably even deprecate the 50%-breaking-tests approach. That will simplify the code. If so, I can take care of removing it (removing code is always a pleasure :D). But well surely it can stay also.

[BlackGad]

No that's not correct I'm afraid. Stryker can only discover if a mutant is covered by a test if a test fails for that mutant. So that means that any test that is failing before we have mutated must be excluded from mutation testing, it cannot give us any results.

Ah base line. Got it :)

[psfinaki]

Right, flaky tests, yeah didn't think about them. Well anyway, even this discussion shows there's not much consensus here so probably we should keep things as they are. Although I guess these two rules can be really just merged into one, having the allowed failing tests percentage specified in the config.

[richardwerkman]

After careful discussion it was decided with richardwerkman (I think) that 50% was as good as any other number.

Yes, we did. It didn't make sense to run stryker with mostly failing tests as the results will for sure be invalid. We could remove the threshold as suggested but I'm not sure what we will gain with that. But now that there is a possibility to break on failing tests I'd be fine with removing the threshold completely if it's what some people want.

But I'm thinking how we would handle 0 succeeding tests. Would we fail or just return a report with a 0 mutation score?

[psfinaki]

IMO, 0 succeeding tests is a very miserable situation and any tool that notices it should cry with pain. Another thing is that 0 mutation score is usually interpreted as "my code is not tested at all" which would not really be the case here.

[rouke-broersma]

I prefer to keep it as it is. We decide what percentage we are comfortable with and can change it at any time if we feel that 50 percent is not correct. If users don't like this behavior they can disable it completely. I don't see much reason to support a custom threshold. I do think we should document this behavior better.