Quality: Add mutation testing to verify test suite effectiveness

[sfloess]

Overview

JNexus has 304 test methods, but mutation testing would verify that these tests actually catch bugs, not just execute code.

What Is Mutation Testing?

Mutation testing introduces bugs (mutations) into code and verifies that tests fail. If tests still pass with a mutation, the tests may be insufficient.

Current State

• 304 test methods across 20 test files
• No mutation testing
• Code coverage unknown (no JaCoCo reports published)

Recommended Tools

For Java (Desktop/Core):
PIT - Industry standard mutation testing

Add to pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.15.8</version>
    <dependencies>
        <dependency>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-junit5-plugin</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <targetClasses>
            <param>org.flossware.jnexus.*</param>
        </targetClasses>
        <targetTests>
            <param>org.flossware.jnexus.*Test</param>
        </targetTests>
        <outputFormats>
            <outputFormat>HTML</outputFormat>
            <outputFormat>XML</outputFormat>
        </outputFormats>
    </configuration>
</plugin>

Run with:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

For Android:

• Same PIT plugin works with Gradle
• Or use Mutant

For iOS:

• Muter for Swift

Example Mutation

Original code:

if (size > maxSize) {
    throw new IllegalArgumentException("Too large");
}

Mutated code (> changed to >=):

if (size >= maxSize) {  // Boundary mutation
    throw new IllegalArgumentException("Too large");
}

If tests still pass, you're missing a boundary test case.

Expected Results

Good mutation score: 75-85% (industry standard)

• Some mutations are equivalent (can't be killed)
• 100% is unrealistic

If score is low (<60%):

• Tests may execute code without asserting behavior
• Missing edge case tests
• Weak assertions

Benefits

• Validates test suite quality
• Finds gaps in edge case testing
• Builds confidence in test coverage
• Prevents false sense of security from high code coverage

Implementation Plan

1. Add PIT to desktop/core builds
2. Run initial baseline report
3. Fix any low-hanging fruit (score <50%)
4. Add to CI (optional - can be slow)
5. Set minimum mutation score threshold

Priority

Low - Nice to have for quality assurance, not blocking

Related

• Testing score: 90/100 (could be 95/100 with mutation testing)