Skip to content

Multi Repo Calibration

Jump to bottom
Cervantes edited this page May 26, 2026 · 7 revisions

Multi-Repo Calibration

Tracking issues:

This page summarizes the current calibration protocol for CervoMutant policy defaults. The source-of-truth document is docs/evaluations/multi-repo-calibration.md in the main repository.

Calibration Targets

Repository Purpose
Cobra Popular CLI library and existing comparison target.
CervoClaw CervoCore Main CervoClaw representative module.
CervoRetry CervoSoft reusable library target.
pflag External popular Go control.

2026-05-26 Sample

Configuration:

  • 20 deterministic mutants per repository and policy.
  • 4 workers.
  • 3 minute budget.
  • Policies: ci-fast and gremlins-compatible.
  • Output root: C:\Users\c___h\AppData\Local\Temp\cervomut-multirepo-calibration-20260526-r2.
Repository Policy/profile Seconds Generated Killed Survived Score Mutation coverage
Cobra ci-fast 6.21 20 13 7 65.00% 100.00%
Cobra gremlins-compatible 6.31 20 13 7 65.00% 100.00%
CervoClaw CervoCore ci-fast 11.51 20 7 13 35.00% 100.00%
CervoClaw CervoCore gremlins-compatible 10.16 20 7 13 35.00% 100.00%
CervoRetry ci-fast 10.41 20 5 15 25.00% 100.00%
CervoRetry gremlins-compatible 5.73 20 5 15 25.00% 100.00%
pflag ci-fast 8.42 20 17 3 85.00% 100.00%
pflag gremlins-compatible 8.31 20 17 3 85.00% 100.00%

Findings

  • ci-fast is stable across the sampled repositories.
  • gremlins-compatible currently matches the same fast operator set, so scores match by design.
  • CervoRetry exposed a Windows junction discovery bug, now fixed in the main repo.
  • CervoCore and CervoRetry have low sampled scores with 100% mutation coverage, which indicates test-suite gaps rather than coverage gaps in this sample.
  • pflag is a useful external control because it is small, popular, and kills most sampled mutants under the same budget.

Next Step

Run 100 deterministic mutants per target, add a nightly sample for CervoCore and CervoRetry, and manually review survivor ranks to tune ranking.

Issue #11 100-Mutant Expansion

Configuration:

  • Policy: ci-balanced.
  • Sampling: deterministic.
  • Cap: 100 mutants.
  • Workers: 2.
Repository Generated Killed Survived Not covered Score Mutation coverage Wall time
Cobra ./doc 100 46 39 15 54.12% 85.00% 31.75s
CervoCore ./... 100 12 28 60 30.00% 40.00% 50.46s
CervoRetry ./... 52 18 10 24 64.29% 53.85% 16.09s
pflag ./... 100 60 14 26 81.08% 74.00% 66.30s

Issue #11 also added:

  • line-range coverage matching from Go coverage profiles;
  • mutation history in .cervomut/history.json;
  • new-vs-long-standing survivor fields in JSON;
  • stricter equivalent-risk suppression rules;
  • assignment-arithmetic and inc-dec as governed default/aggressive operators;
  • stronger CI smoke gates for mutator registry, ci-fast, and ci-balanced.

Implemented After This Sample

The next implementation cycle applied the non-calibration improvements before expanding to larger samples:

  • cervomut fast ./... is now the Gremlins-like fast path.
  • ci-fast and ci-balanced write summary, JSON, and JUnit by default; HTML is reserved for nightly, campaign, or explicit config.
  • JSON results now include selection_reason, coverage_source, survivor_rank, rank_score, actionability, suggested_test_scope, and nearest_tests.
  • Summary reports include equivalent-risk statistics and suppression audit counts.
  • Operator definitions now declare compile-error risk and inclusion reason.
  • New governed operators were added: string-empty-literals, numeric-literals, return-bool-literals, slice-map-len-boundary, plus generated loop-control.
  • Inline ignore parsing now only treats comment directives as ignores, not string literals containing cervomut:ignore.

Clone this wiki locally