Comparison Harness

Comparison Harness

Tracking issue: https://github.com/cervantesh/cervo-mutants/issues/13

The comparison harness runs CervoMutant beside external Go mutation-testing tools while preserving target semantics, memory limits, logs, partial reports, and denominator health.

Source files in the main repository:

File	Role
docs/evaluations/go-repo-pool-40.json	Repository manifest with name, URL, target, lane, domain, and reason.
scripts/compare-tools-pool.ps1	Main multi-tool runner for Windows/PowerShell.
scripts/calibration-smoke.ps1	Lighter CervoMutant-only calibration smoke runner.
cmd/cervomut compare	Normalizes existing tool reports into one JSON schema.
pkg/extcompare	Parser and comparability logic for CervoMutant, Gremlins, gomu, and go-mutesting reports.

Recommended Gremlins Comparison

For CervoMutant vs Gremlins over the current 10 small repositories:

.\scripts\compare-tools-pool.ps1 `
  -Tools cervomut,gremlins `
  -Names cobra,pflag,logrus,uuid,decimal,gjson,sjson,jsonparser,burntsushi-toml,urfave-cli `
  -CompareTargetMode package-root `
  -GremlinsTargetMode package-root `
  -Workers 2 `
  -TimeoutSeconds 600 `
  -MinFreeMemoryMB 8192 `
  -MinFreeCommitMB 8192 `
  -KillBelowFreeMemoryMB 4096 `
  -KillBelowFreeCommitMB 4096 `
  -MaxProcessTreeMemoryMB 6144 `
  -MemoryPollSeconds 1 `
  -GoMemoryLimit 3GiB `
  -GoMaxProcs 2 `
  -GoFlags "-p=2" `
  -Resume

This makes every selected tool receive the same effective target. For example, when the manifest target is ./..., package-root mode passes . to the tools.

CervoMutant Command Shape

The harness runs CervoMutant through the bounded comparison lane:

cervomut run <effective_target> --policy comparison-safe --workers <n> --out <repoOut>/cervomut

comparison-safe uses:

• gremlins-compatible operators;
• overlay isolation;
• deterministic sampling;
• max 2 workers;
• 10 minute run budget;
• 20 second per-mutant timeout;
• default 250 mutant cap when no explicit cap is set.

External Tool Command Shapes

Tool	Command shape
Gremlins	gremlins unleash <effective_target> --workers <n> --threshold-efficacy 0 --threshold-mcover 0 --output <repoOut>/gremlins.json
gomu	gomu run <effective_target> --workers <n> --timeout <seconds> --threshold 0 --fail-on-gate=false --output json
go-mutesting	go-mutesting /noop /quiet /no-diffs /logger-summary-json /logger-agentic-json /exec-timeout:<seconds> /workers:<n> <effective_target>

Memory And Timeout Guards

Important knobs:

• TimeoutSeconds: per tool/repo wall-clock limit.
• MinFreeMemoryMB: wait before starting if physical memory is too low.
• MinFreeCommitMB: wait before starting if commit headroom is too low.
• KillBelowFreeMemoryMB: kill a running process if free memory drops below this value.
• KillBelowFreeCommitMB: kill a running process if commit headroom drops below this value.
• MaxProcessTreeMemoryMB: kill if the process tree exceeds this private or working-set memory.
• MemoryPollSeconds: polling interval.
• GoMemoryLimit, GoMaxProcs, GoFlags: Go runtime controls applied to the child process.

Use low workers and a process-tree memory cap for local Windows runs.

Outputs

Each run writes:

• summary.json at OutputRoot;
• per-repo logs under OutputRoot/<repo>/<tool>.log;
• CervoMutant reports under OutputRoot/<repo>/cervomut;
• Gremlins JSON under OutputRoot/<repo>/gremlins.json;
• copied gomu/go-mutesting reports when available.

For CervoMutant, inspect in this order:

1. mutation-report.json
2. partial-summary.json
3. partial-mutation-report.json

When the final report is missing, the harness falls back to partial reports and sets partial_report_used=true.

Status Values

The harness uses actionable statuses:

• ok
• timeout
• partial_timeout
• watchdog_kill
• partial_watchdog_kill
• skipped
• panic
• no_report
• no_results
• all_timed_out
• not_covered_only

Agent Checklist

Before launching a comparison:

1. Confirm the GitHub issue.
2. Choose manifest or package-root mode.
3. Use the same effective target mode for every tool in any fairness claim.
4. Set memory and timeout guards.
5. Record tool binary paths and versions when possible.
6. Use -Resume for long pools.

After the run:

1. Inspect summary.json.
2. Check apples_to_apples_key before comparing score or speed.
3. Inspect logs for panic, No results to report, timeout, or watchdog kill.
4. Prefer final reports, then partial summaries, then full partial reports.
5. Document denominator health warnings, not just score.
6. Update issue #13 and the findings ledger.