Skip to content

ci(mutation): run as scheduled audit + opt-in instead of on every push to main#787

Merged
grunch merged 1 commit into
mainfrom
ci/mutation-schedule-only
Jun 19, 2026
Merged

ci(mutation): run as scheduled audit + opt-in instead of on every push to main#787
grunch merged 1 commit into
mainfrom
ci/mutation-schedule-only

Conversation

@grunch

@grunch grunch commented Jun 19, 2026

Copy link
Copy Markdown
Member

What

Stop running full mutation testing on every push to main. Keep it as a periodic + on-demand audit and an opt-in per-PR job, and speed up the tool install.

Why

The mutation-baseline job ran a full cargo mutants over the whole codebase on every push to main (the run that prompted this: it sat in progress for 20+ min). Two problems:

  • It is continue-on-error: true, so it gates nothing — it only produces a report.
  • Mutation testing is inherently slow (one full test-suite run per mutant), so a per-merge full run burns a lot of CI minutes for no enforcement value, especially when a weekly scheduled run already exists.

Moving it fully local was considered and rejected: a non-gating, informational job that depends on each dev remembering to run a slow command locally would simply rot, and we'd lose the published HTML report and historical artifacts. The right fix is changing when CI runs it, not removing it.

Changes

  • Drop the push: [main] trigger. The baseline now runs via:
    • schedule — weekly full run (unchanged), HTML report to GitHub Pages
    • workflow_dispatch — manual on-demand full run
    • opt-in per-PR job via the existing run-mutation label (unchanged)
  • Swap cargo install cargo-mutantstaiki-e/install-action (prebuilt binaries) in both jobs, saving the from-source compile each run.
  • Docs (docs/MUTATION_TESTING.md): describe the new triggers and point to cargo mutants --in-diff for pre-push local feedback.

Notes

  • No behavior change for the actual mutation analysis — only its triggers and install method.
  • YAML validated.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Updated mutation testing documentation with new workflow descriptions and local pre-push testing command example.
  • Chores

    • Modified mutation testing workflow to require opt-in via label for pull requests, while maintaining weekly scheduled and manual on-demand runs.

Full mutation testing is slow (one test-suite run per mutant) and the job is
`continue-on-error`, so running the baseline on every push to `main` gated
nothing and only burned CI minutes. Drop the `push: [main]` trigger; the
baseline now runs weekly (`schedule`) and on demand (`workflow_dispatch`),
alongside the existing opt-in per-PR job (`run-mutation` label).

Also swap `cargo install cargo-mutants` (compiles from source on every run)
for `taiki-e/install-action` (prebuilt binaries) in both jobs.

Docs updated to describe the new triggers and point to `cargo mutants
--in-diff` for pre-push feedback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126WFgFsxsvCewFb9pxfT7v
@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The mutation testing workflow removes the push trigger for the main branch and replaces inline cargo install cargo-mutants steps with the taiki-e/install-action@v2 action in both the PR and baseline jobs. Documentation is updated to reflect the new opt-in label, weekly schedule, and workflow_dispatch trigger model, and adds a pre-push local command example.

Changes

Mutation Testing CI and Docs Update

Layer / File(s) Summary
Workflow trigger and install action changes
.github/workflows/mutation.yml
Removes push trigger for main, adds comments stating mutation testing is intentionally not run on push, and replaces cargo install cargo-mutants with taiki-e/install-action@v2 (tool: cargo-mutants) in both the PR mutation job and the baseline main job.
Documentation updates
docs/MUTATION_TESTING.md
Updates CI/CD section to describe opt-in PR label, weekly scheduled HTML report to GitHub Pages, workflow_dispatch, and continue-on-error behavior; adds a pre-push local cargo mutants --in-diff command example.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~4 minutes

Possibly related PRs

  • MostroP2P/mostro#619: Originally introduced the mutation testing infrastructure in .github/workflows/mutation.yml and docs/MUTATION_TESTING.md that this PR modifies.
  • MostroP2P/mostro#648: Previously added the opt-in run-mutation label logic for PR mutation testing, which this PR extends by removing the push-to-main trigger.

Suggested reviewers

  • arkanoider

Poem

🐇 No more mutations on every main push,
A label opt-in gives the workflow a hush.
install-action now fetches the tool,
Weekly schedules keep testing cool.
Docs updated, the rabbit hops free! 🌿

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: mutation testing is shifted from running on every push to main to running on a schedule and with opt-in per-PR via labels.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/mutation-schedule-only

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/mutation.yml (1)

44-46: ⚡ Quick win

Consider pinning the action to a specific commit SHA.

While using @v2 simplifies maintenance, pinning to a specific SHA (e.g., taiki-e/install-action@3068b918e10e4f9f92c7f333305bf152ad1e03b2) improves supply-chain security and build reproducibility. Many projects accept the @v2 tradeoff for easier updates, but if this repository has a blanket pinning policy, you should pin this action.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/mutation.yml around lines 44 - 46, The
taiki-e/install-action currently uses `@v2` version reference which is not pinned
to a specific commit SHA. Update the uses statement for taiki-e/install-action
to reference a specific commit SHA instead of the version tag to improve
supply-chain security and build reproducibility. Replace `@v2` with the full
commit SHA format as indicated in the review comment.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/mutation.yml:
- Around line 44-46: The taiki-e/install-action currently uses `@v2` version
reference which is not pinned to a specific commit SHA. Update the uses
statement for taiki-e/install-action to reference a specific commit SHA instead
of the version tag to improve supply-chain security and build reproducibility.
Replace `@v2` with the full commit SHA format as indicated in the review comment.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 28c30e1d-651d-4391-9102-c8f222711631

📥 Commits

Reviewing files that changed from the base of the PR and between 6802c97 and 4f25a36.

📒 Files selected for processing (2)
  • .github/workflows/mutation.yml
  • docs/MUTATION_TESTING.md

@grunch grunch merged commit f64e93b into main Jun 19, 2026
8 checks passed
@grunch grunch deleted the ci/mutation-schedule-only branch June 19, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant