Test Impact Analysis: Code coverage based selective test-rerunning

[tothambrus11]

It would be interesting to explore techniques for prioritizing and potentially also filtering tests based on:

• what code area each test case covered last time it was run
• what the code diffs since the run are
• how long each test case took to run.

By prioritizing cases that were most affected by our latest changes, and within those, we still prioritize faster tests, we can significantly reduce the time it takes to see if something has failed. Additionally, we may be able to prove that the changes we did could not possibly affect particular test cases, so we can skip rerunning them.

The question is whether there are sound techniques to do this over time, assuming the tests don't rely on significant side-effects.

[tothambrus11]

It turns out, this is called test impact analysis: https://martinfowler.com/articles/rise-test-impact-analysis.html

We should look at this when designing Hylo's testing framework.

[tothambrus11]

Prioritizing Test Gaps by Risk in Industrial Practice -
An Automated Approach and Multimethod Study: https://www.se.cs.uni-saarland.de/publications/docs/HSJ+25.pdf