Backtrack tests which test concrete part

[HonzaMac]

Question

Stryker has functionality to do test coverage analysis for tests and code.
Would it be possible to get information which line of code is tested by which tests?

My problem is, that I have more than 2000 tests and a lot of them test same parts.
I would like to reduce/delete those tests to gain speed and better maintainability of whole test suite.

[nicojs]

Wow, that is an awesome idea!

How would you like this feature to work? We've been thinking of making the tests visible that cover a specific mutant if you click on it in the HTML report, but that would still be a lot of work for the developer.

Test coverage analysis is a powerful thing with a lot of potential. We should get a lot more out of it than we do right now.

[HonzaMac]

How would you like this feature to work? We've been thinking of making the tests visible that cover a specific mutant if you click on it in the HTML report, but that would still be a lot of work for the developer.

This would be great first step. I would immediately know, where to go to improve coverage.

Stryker is tool for testing test suite, having view on test suite make sense as well.

I am thinking more about improving test report in this format (i.e. Mocha output):

Basic functionality (killed 32, survived 8)
  concrete functionality (killed 16, survived 7)
    ✓ should report test as successful (killed 10, survived 2) 
    ✓ should report test as failed (killed 4, survived 5)
    ✓ should kill the child process (killed 2, survived 0)
  duplicate concrete functonality (killed 16, survived 1)
    ✓ should report mutant as survived (killed 10, survived 0)
    ✓ should report mutant as killed (killed 6, survived 1)

5 passing (33s)

I can clearly see, which test case I should adjust to kill more mutants.
I also see which test suite can be dropped.

I am thinking about "Redundancy Score".
If test have some overlap with other test, then it is redundant. If same mutation was killed by more than one test, it is kind of redundant. If some test miss mutant it is also missed opportunity for test case/suite.

[nicojs]

This would be a great first step. I would immediately know, where to go to improve coverage.

For that to work, it should be supported by the HTML reporter. The HTML reporter is housed in its GH repo. Could you please add a remark here on how you want this to look? Right now we have a modal dialog, but adding all kind of info in a modal dialog might not be the most user-friendly.

stryker-mutator/mutation-testing-elements#51

If test have some overlap with other test, then it is redundant. If same mutation was killed by more than one test, it is kind of redundant. If some test miss mutant it is also missed opportunity for test case/suite.

That is an interesting idea. There is 1 caveat: right now we're running each test suite in --bail mode, meaning that we only know the first test that killed a specific mutant, more tests might fail but we don't know about them. This is great for performance, but might not be what you expect. I think your example of the mocha output is a great idea, but the "killed" amounts will be a lot less than in your example. Ideally, you would want to see "1 killed" for tests if you're practicing TDD and writing pure unit tests. You would especially be interested in tests that don't have killed mutants, probably your tests have overlap.

[nicojs]

Closed with #2868