Compute overlap between the generated tests to generate fewer tests

[vmassol]

Right now DSpot generate new tests independently of one another, i.e. each test is generated and have its mutation score compared to the original baseline and if the mutation score is greater then the new test is kept. This leads to having a lot of new tests added that are the same (see below).

One idea would be that once DSpot succeeds in generating a new test, this new test is added to the baseline and thus new tests need to increase the mutation score of that new baseline to not be discarded.

For example 2 new tests have been generated in on some XWiki module:

In ProviderTest:

    public void loadAndInjectProviderWhenExceptionInInitialize() throws Exception {
        EmbeddableComponentManager cm = new EmbeddableComponentManager();
        Assert.assertNull(((EmbeddableComponentManager) (cm)).getParent());
        Assert.assertNull(((EmbeddableComponentManager) (cm)).getNamespace());
        Assert.assertNull(((EmbeddableComponentManager) (cm)).getComponentEventManager());
        cm.initialize(getClass().getClassLoader());
...

In EmbeddableComponentManagerTest:

    @Test(timeout = 10000)
    public void testUnregisterDisposableSingletonComponentWithInstance() throws Exception {
        EmbeddableComponentManager ecm = new EmbeddableComponentManager();
        Assert.assertNull(((EmbeddableComponentManager) (ecm)).getParent());
        Assert.assertNull(((EmbeddableComponentManager) (ecm)).getComponentEventManager());
        Assert.assertNull(((EmbeddableComponentManager) (ecm)).getNamespace());
        DefaultComponentDescriptor<EmbeddableComponentManagerTest.Role> cd = new DefaultComponentDescriptor<EmbeddableComponentManagerTest.Role>();
...

In practice we don't need that DSpot generates new tests wherever it finds EmbeddableComponentManager ecm = new EmbeddableComponentManager(); by adding the 3 asserts... This won't increase the test coverage nor the mutation score. But it will increase the test execution time if we want to include automatically all the tests that DSpot generates.

Thus it would be nice to have an option for this :)

Thx

[andrewbwogi]

Hi @vmassol, I will have a look at this.

[vmassol]

Thanks @andrewbwogi

[danglotb]

Hello @andrewbwogi, any news on this? :-)

[andrewbwogi]

Hi @danglotb, I first tried to investigate if we can control better how amplified tests are generated to avoid producing identical tests but it seems like a too complex task for now. The original suggestion by @vmassol to add improved scores to the baseline is a simpler approach.

I've modified the PitMutantScore, JacocoCoverage and CloverCoverage selectors, the ChangeDetector and testing the modifications remain.

[danglotb]

That's great news! Thank you very much, I cannot wait to see your pull request.

[monperrus]

@andrewbwogi any news on this?

[andrewbwogi]

Hi @monperrus, I'm currently testing modifications I've made on PitMutantScore, JacocoCoverage and CloverCoverage selectors. For the ChangeDetector selector it seems that new tests must be syntactically compared to previously selected tests since there is no score in that selector that can be added to a baseline. I'm investigating if I can use Spoon patterns to match new tests with previously selected tests.

[andrewbwogi]

I've looked at pattern matching in Spoon and it can be used to find syntax equality if the compared methods have the exact same AST, including variable names. An expanded set of syntax equalities can be found if variable names are systematically replaced so that a test containing only:

EmbeddableComponentManager cm = new EmbeddableComponentManager();
Assert.assertNull(((EmbeddableComponentManager) (cm)).getParent());

would be considered equal to a test containing only:

EmbeddableComponentManager ecm = new EmbeddableComponentManager();
Assert.assertNull(((EmbeddableComponentManager) (ecm)).getParent());

A further simplification of generated tests could be achieved by removing any assertion that already exists in a previously generated test and has the same side effecting AST leading up to it as in the previously generated test. This would take care of some cases where a new test AST is a subtree of another new test AST.

@vmassol and @danglotb, can you tell if these modifications will be relevant for a significant amount of generated tests? Is it worth making these adjustments to the Change Detector Selector, considering how often it is used?

[vmassol]

can you tell if these modifications will be relevant for a significant amount of generated tests? Is it worth making these adjustments to the Change Detector Selector, considering how often it is used?

@andrewbwogi It's hard for me to judge knowing that, as of today I'm not able to get DSpot to generate lots of tests on XWiki (not been able to generate any tests with amplifiers so far). The only few tests that I was able to generate are when there are no amplifiers and there aren't enough IMO to draw conclusions on this point.

[andrewbwogi]

Here is a pull request with fixes for Pit and Clover selectors: #726. @vmassol, can you check if that pull request fixes your example?

[danglotb]

@andrewbwogi thank you very much for the pull request.

However, since the CloverSelector has been removed since it was unstable and redundant with JacocoSelector, a part of your changes won't be necessary.

I'm gonna review the changes ASAP.

Thank you!