Multi Repo Calibration

Multi-Repo Calibration

Tracking issue: https://gitea.cervbox.synology.me/CervoSoft/cervo-mutant/issues/10

This page summarizes the current calibration protocol for CervoMutant policy defaults. The source-of-truth document is docs/evaluations/multi-repo-calibration.md in the main repository.

Calibration Targets

Repository	Purpose
Cobra	Popular CLI library and existing comparison target.
CervoClaw CervoCore	Main CervoClaw representative module.
CervoRetry	CervoSoft reusable library target.
pflag	External popular Go control.

2026-05-26 Sample

Configuration:

• 20 deterministic mutants per repository and policy.
• 4 workers.
• 3 minute budget.
• Policies: ci-fast and gremlins-compatible.
• Output root: C:\Users\c___h\AppData\Local\Temp\cervomut-multirepo-calibration-20260526-r2.

Repository	Policy/profile	Seconds	Generated	Killed	Survived	Score	Mutation coverage
Cobra	ci-fast	6.21	20	13	7	65.00%	100.00%
Cobra	gremlins-compatible	6.31	20	13	7	65.00%	100.00%
CervoClaw CervoCore	ci-fast	11.51	20	7	13	35.00%	100.00%
CervoClaw CervoCore	gremlins-compatible	10.16	20	7	13	35.00%	100.00%
CervoRetry	ci-fast	10.41	20	5	15	25.00%	100.00%
CervoRetry	gremlins-compatible	5.73	20	5	15	25.00%	100.00%
pflag	ci-fast	8.42	20	17	3	85.00%	100.00%
pflag	gremlins-compatible	8.31	20	17	3	85.00%	100.00%

Findings

• ci-fast is stable across the sampled repositories.
• gremlins-compatible currently matches the same fast operator set, so scores match by design.
• CervoRetry exposed a Windows junction discovery bug, now fixed in the main repo.
• CervoCore and CervoRetry have low sampled scores with 100% mutation coverage, which indicates test-suite gaps rather than coverage gaps in this sample.
• pflag is a useful external control because it is small, popular, and kills most sampled mutants under the same budget.

Next Step

Run 100 deterministic mutants per target, add a nightly sample for CervoCore and CervoRetry, and manually review survivor ranks to tune ranking.