Skip to content

Multi Repo Calibration

Jump to bottom
Cervantes edited this page Jun 17, 2026 · 7 revisions

Multi-Repo Calibration

Tracking issues:

This page summarizes the current calibration protocol for CervoMutant policy defaults. The source-of-truth document is docs/evaluations/multi-repo-calibration.md in the main repository.

For external tool comparisons, follow:

Do not summarize CervoMutant vs Gremlins, gomu, or go-mutesting as a fairness claim unless compared rows share the same effective_target and target_mode.

Calibration Targets

Repository Purpose
Cobra Popular CLI library and existing comparison target.
First-party application core Main first-party application representative module.
First-party shared library Reusable library target.
pflag External popular Go control.

2026-05-26 Sample

Configuration:

  • 20 deterministic mutants per repository and policy.
  • 4 workers.
  • 3 minute budget.
  • Policies: ci-fast and gremlins-compatible.
  • Output root: C:\Users\c___h\AppData\Local\Temp\cervomut-multirepo-calibration-20260526-r2.
Repository Policy/profile Seconds Generated Killed Survived Score Mutation coverage
Cobra ci-fast 6.21 20 13 7 65.00% 100.00%
Cobra gremlins-compatible 6.31 20 13 7 65.00% 100.00%
First-party application core ci-fast 11.51 20 7 13 35.00% 100.00%
First-party application core gremlins-compatible 10.16 20 7 13 35.00% 100.00%
First-party shared library ci-fast 10.41 20 5 15 25.00% 100.00%
First-party shared library gremlins-compatible 5.73 20 5 15 25.00% 100.00%
pflag ci-fast 8.42 20 17 3 85.00% 100.00%
pflag gremlins-compatible 8.31 20 17 3 85.00% 100.00%

Findings

  • ci-fast is stable across the sampled repositories.
  • gremlins-compatible currently matches the same fast operator set, so scores match by design.
  • The first-party shared library exposed a Windows junction discovery bug, now fixed in the main repo.
  • The first-party application core and shared library have low sampled scores with 100% mutation coverage, which indicates test-suite gaps rather than coverage gaps in this sample.
  • pflag is a useful external control because it is small, popular, and kills most sampled mutants under the same budget.

Next Step

Run 100 deterministic mutants per target, add a nightly sample for the first-party application core and shared library, and manually review survivor ranks to tune ranking.

Issue #11 100-Mutant Expansion

Configuration:

  • Policy: ci-balanced.
  • Sampling: deterministic.
  • Cap: 100 mutants.
  • Workers: 2.
Repository Generated Killed Survived Not covered Score Mutation coverage Wall time
Cobra ./doc 100 46 39 15 54.12% 85.00% 31.75s
First-party application core ./... 100 12 28 60 30.00% 40.00% 50.46s
First-party shared library ./... 52 18 10 24 64.29% 53.85% 16.09s
pflag ./... 100 60 14 26 81.08% 74.00% 66.30s

Issue #11 also added:

  • line-range coverage matching from Go coverage profiles;
  • mutation history in .cervomut/history.json;
  • new-vs-long-standing survivor fields in JSON;
  • stricter equivalent-risk suppression rules;
  • assignment-arithmetic and inc-dec as governed default/aggressive operators;
  • stronger CI smoke gates for mutator registry, ci-fast, and ci-balanced.

Implemented After This Sample

The next implementation cycle applied the non-calibration improvements before expanding to larger samples:

  • cervomut fast ./... is now the Gremlins-like fast path.
  • ci-fast and ci-balanced write summary, JSON, and JUnit by default; HTML is reserved for nightly, campaign, or explicit config.
  • JSON results now include selection_reason, coverage_source, survivor_rank, rank_score, actionability, suggested_test_scope, and nearest_tests.
  • Summary reports include equivalent-risk statistics and suppression audit counts.
  • Operator definitions now declare compile-error risk and inclusion reason.
  • New governed operators were added: string-empty-literals, numeric-literals, return-bool-literals, slice-map-len-boundary, plus generated loop-control.
  • Inline ignore parsing now only treats comment directives as ignores, not string literals containing cervomut:ignore.

Clone this wiki locally