Tool Comparison Protocol

Tool Comparison Protocol

Tracking issue: https://github.com/cervantesh/cervo-mutants/issues/13

This protocol prevents invalid mutation-testing claims such as comparing cervomut run ./... with gremlins unleash . as if both commands covered the same scope.

Required Fields

Every comparison row must record:

• target: the manifest target requested by the study, for example ./....
• effective_target: the target actually passed to the tool, for example ..
• target_mode: the normalization strategy, usually manifest or package-root.
• manifest_equivalent: whether the effective target is the same scope as the manifest target.
• apples_to_apples_key: stable grouping key made from target mode and effective target.

A comparison is apples-to-apples only when rows share the same effective_target and target_mode.

Study Modes

Manifest Mode

Use this to test whether each tool can run the manifest target as written.

manifest target: ./...
CervoMutant effective target: ./...
Gremlins effective target: ./...
gomu effective target: ./...
go-mutesting effective target: ./...
target_mode: manifest

If a tool cannot interpret the manifest target correctly, record that as an operational finding. Do not silently normalize only that tool.

Package-Root Mode

Use this for fair package-root comparisons, especially against Gremlins.

manifest target: ./...
CervoMutant effective target: .
Gremlins effective target: .
gomu effective target: .
go-mutesting effective target: .
target_mode: package-root
manifest_equivalent: false

This is fair between tools, but it is not equivalent to a full ./... module campaign.

Diagnostic Mode

Use diagnostic mode only to understand tool behavior.

CervoMutant effective target: ./...
Gremlins effective target: .

Diagnostic results must not be summarized as "Tool A is faster/better than Tool B" without a target-semantics warning.

Required Metrics

Every result must carry denominator components:

• total/generated mutants;
• killed;
• survived;
• not covered;
• timed out;
• errors or not viable mutants;
• test efficacy over killed plus survived;
• mutation coverage where available;
• denominator health warnings.

Raw mutation score is not enough. A high score with a tiny effective denominator and many timed-out or not-covered mutants is unhealthy.

Reporting Labels

Every comparison summary must include one of:

• apples_to_apples=true: same effective target and target mode.
• manifest_equivalent=false: comparable between tools, but not the original manifest scope.
• not_comparable: effective targets or target modes differ.

When in doubt, mark the row not comparable and explain why.