Update mutation testing ADR and execution plan with change-detection

[leynos]

Summary

• Introduces a daily, change-detection gated mutation-testing workflow. Heavy mutation runs are only executed when Rust source files changed on origin/main within the last 24 hours.
• Mutations are scoped to changed files, with separate invocations for the root crate and the non-workspace member wireframe_testing.
• Updates ADR and execution plan docs to reflect the new approach, including corrected jq filters and explicit change-detection behavior.

Changes

• ADR update: docs/adr-007-mutation-testing-with-cargo-mutants.md

  • Date and decision outcomes updated.
  • Schedule changed from weekly to daily; introduced a dedicated change-detection section.
  • Defined code-change patterns to monitor and clarified manifest-only changes handling.
  • Documented workspace split handling (root crate vs wireframe_testing) and manual dispatch bypass.
  • Added a Summary of resolved decisions and corrected jq field references.

• Execution plan update: docs/execplans/mutation-testing.md

  • Replaced previous weekly schedule with daily schedule guarded by change-detection.
  • Added Detect changed Rust files step (commit-timestamp based) and three-step gating:
    1. has_changes flag, 2) root_files, 3) wt_files outputs.
  • Described two targeted mutation runs:
    • Root crate: cargo mutants with --file per changed root file.
    • wireframe_testing: cargo mutants --dir wireframe_testing with --file per changed file.
  • Artefacts: separate artefacts for root and wireframe_testing reports when applicable.
  • Summary: enhanced post-processing with corrected jq filters and a per-report post_results function.
  • Updated risk/constraints, decisions, and validation notes.

• Workflow and gate behavior (documentation sketch updated):

  • Triggers: daily cron (cron: "30 4 * * *") and workflow_dispatch for ad-hoc runs.
  • fetch-depth: 0 for full history in change-detection.
  • Heavy steps gated by has_changes: Setup Rust, Install cargo-mutants, and Run mutation testing only run when changes exist.
  • Separate root vs wireframe_testing mutation runs with appropriate file/dir scoping.
  • Separate artefact uploads and per-report summaries.
  • Post-summary updated to reflect per-crate results and surviving mutants with correct field references.

Why this change

• Mutation testing is expensive; validating it daily with a guard that only runs when relevant code changed significantly reduces cost while preserving feedback speed.
• Splitting runs by crate improves coverage for non-workspace components and ensures all Rust sources are tested as applicable.
• Clear, explicit monitoring of changes reduces noise from manifest-only changes and supports faster CI feedback loops.

Known Risks and Mitigations

• Execution time: gating reduces cost, but large, widespread changes can still take time. Parallelism may help but increases resource usage.
• False survivors: still require human triage; this ensures artifacts and summaries are actionable.
• Runner cost: daily cadence increases total minutes, but most days will exit quickly due to no changes.
• Change-detection edge cases: relies on commit timestamps (not reflog); manual workflow_dispatch bypass provides a fallback.
• Manifest-only changes: excluded from automatic mutation runs; manual dispatch recommended if they may affect outcomes.
• Tool stability: cargo-mutants is not yet 1.0; jq filters must be validated against the tool's output and updated if needed.

Resolved Decisions

• Scope filtering via per-file invocations for both root and wireframe_testing.
• Workspace split handling for wireframe_testing (not a workspace member).
• Schedule cadence updated to daily with a change-detection guard.
• Change-detection mechanism using git log --since="24 hours ago" with commit timestamps.
• Code-change definitions monitor src//*.rs, wireframe_testing/src//.rs, examples/**/.rs, benches/**/*.rs; manifest changes ignored.
• Manual dispatch bypass remains for ad-hoc validation or manifest changes.

Test plan

• Trigger the daily workflow on main with no relevant changes: expect a fast skip message in the job summary and no heavy mutation runs.
• Make a small Rust change in a monitored path: expect two mutations runs (root and possibly wireframe_testing) with appropriate --file/--dir args and separate artefacts; verify summary shows counts and a table of survivors if any.
• Trigger a manual workflow_dispatch run: verify guard bypasses change-detection and mutations run unconditionally.
• Validate the final job summary formatting and per-mutant listing; ensure artefacts are downloadable and aligned with the summary.

Documentation impact

• ADR and exec plan docs updated to reflect daily cadence, guarded execution, and per-crate mutation runs.
• Corrected jq field names and references in summaries.

Notes for reviewers

• Pay attention to the correctness of the change-detection logic and file-pattern wiring described in the ADR and exec plan.
• Verify that the proposed jq filters align with the actual cargo-mutants JSON schema and the updated post_results function.
• Confirm that the new workflow sketch aligns with the ADR updates and the intended automation behavior.

◳ Generated by DevBoxer ◰

---

ℹ️ Tag @devboxerhub to ask questions and address PR feedback

📎 Task: https://www.devboxer.com/task/a05ecdfd-5675-48b1-91a0-31dc3d90de87

Summary by Sourcery

Documentation:

• Document daily, change-detection–guarded mutation testing runs with per-file scoping and separate root vs wireframe_testing invocations in the ADR and execution plan.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6a567743-e7f7-43fb-b1ef-ea8d607e0974

📥 Commits

Reviewing files that changed from the base of the PR and between dc68464 and 0226472.

📒 Files selected for processing (1)

• docs/execplans/mutation-testing.md

 __________________________________________________________________________________________________________________________________________
< Test early. Test often. Test automatically. Tests that run with every build are much more effective than test plans that sit on a shelf. >
 ------------------------------------------------------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

Walkthrough

Summarise the documentation and workflow updates: scheduling moved from weekly to daily; add a 24‑hour git-based Rust-file change detector that gates CI; install/run steps run only when changes detected (or on manual dispatch); mutation runs split into root and wireframe_testing, scoped to changed files; artifacts and summary generation adjusted accordingly.

Changes

Cohort / File(s)	Summary
Mutation Testing ADR docs/adr-007-mutation-testing-with-cargo-mutants.md	Updated ADR date/formatting and CI behaviour: cron changed to daily; added git log --since="24 hours ago" change detection to set has_changes and short-circuit when false; conditional setup/install/run steps; split mutation execution into scoped root and wireframe_testing invocations with repeated --file args; separate artefact dirs and conditional summary posting.
Execution Plan / How‑to docs/execplans/mutation-testing.md	Documented daily schedule, explicit fetch-depth: 0 requirement, change-detection patterns (exclude manifest-only changes; monitor src/**/*.rs, wireframe_testing/src/**/*.rs, examples/**/*.rs, benches/**/*.rs), gated install/run steps, per-crate mutants.out/ and mutants-wt.out/ handling, and updated jq snippets for "$dir/outcomes.json" reading and conditional summary logic.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Scheduler as Scheduler (cron / dispatch)
  participant Actions as GitHub Actions runner
  participant Git as git (repo)
  participant Installer as Rust + cargo-mutants installer
  participant MutantsRoot as cargo-mutants (root)
  participant MutantsWT as cargo-mutants (wireframe_testing)
  participant Storage as Artifacts / Summary

  Scheduler->>Actions: trigger workflow
  Actions->>Git: checkout (fetch-depth: 0)
  Actions->>Git: run git log --since="24 hours ago" for `.rs` paths
  alt has_changes = false
    Actions->>Storage: append skip message to $GITHUB_STEP_SUMMARY
    Actions->>Actions: finish job (no install/run)
  else has_changes = true
    Actions->>Installer: install Rust toolchain & cargo-mutants
    Installer->>MutantsRoot: run cargo mutants --file ... (scoped to changed files)
    MutantsRoot->>Storage: upload `mutants.out/` and outcomes.json
    Installer->>MutantsWT: run cargo mutants --dir wireframe_testing --file ... (scoped)
    MutantsWT->>Storage: upload `mutants-wt.out/` and outcomes.json
    Actions->>Storage: generate summary by reading outcomes.json from each dir
  end

Loading

Possibly Related PRs

• Add mutation testing ADR and execplan #512 — Modifies the same mutation-testing ADR and execplan; provides the earlier ADR/execplan that this change updates to daily scheduling and gated, per-crate runs.

Poem

🧬 Daily checks now watch the code,
Skip the run if nothing showed,
Scope the mutants to the files,
Split the runs and store the piles,
CI hums efficient, bright and bold.

🚥 Pre-merge checks | ✅ 5 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Developer Documentation	⚠️ Warning	The PR introduces new tooling and build requirements (cargo-mutants workflow, change-detection mechanisms) documented in the ADR and execution plan but not in docs/developers-guide.md as required.	Update docs/developers-guide.md to add a section documenting the cargo-mutants mutation testing tooling requirement, including workflow details and developer guidance.
User-Facing Documentation	❓ Inconclusive	The user has provided shell commands and instructions for examining git changes, but no actual verification data or context about what is being checked.	Provide the actual changed files, their content, and the specific verification criteria you need assessed.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly summarises the main changes: updating mutation testing ADR and execution plan documentation with change-detection logic.
Description check	✅ Passed	The description is comprehensive and clearly related to the changeset, detailing the introduction of daily change-detection–gated mutation testing, scoped mutations, and updated documentation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Testing	✅ Passed	Pull request contains exclusively documentation and CI workflow configuration changes with no new executable functionality.
Module-Level Documentation	✅ Passed	The custom check for module-level documentation applies exclusively to code modules. This pull request modifies only Markdown documentation files with no changes to any code modules.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch add-change-detection-to-mutation-testing-691oqr

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Reviewer's Guide

Documents a revised, daily mutation-testing workflow that is gated by change detection, scopes runs to changed Rust files, splits execution between the root crate and the non-workspace wireframe_testing crate, and fixes jq-based reporting in both the ADR and execution plan.

Sequence diagram for the daily, change-detection–gated mutation testing workflow

sequenceDiagram
    actor Developer
    participant GitHubActions
    participant Checkout
    participant DetectStep
    participant SetupRust
    participant InstallCargoMutants
    participant RootMutantsRun
    participant WTMutantsRun
    participant UploadArtifacts
    participant PostSummary

    Developer->>GitHubActions: Trigger workflow_dispatch (optional)
    GitHubActions->>GitHubActions: Daily cron trigger

    GitHubActions->>Checkout: actions_checkout with ref (branch or main)
    Checkout-->>GitHubActions: Repo at selected ref (fetch-depth 0)

    GitHubActions->>DetectStep: Run git log --since="24 hours ago" origin/main -- '*.rs'
    DetectStep-->>DetectStep: Split into root_files and wt_files
    DetectStep-->>GitHubActions: has_changes, root_files, wt_files outputs

    alt No relevant Rust changes
        GitHubActions->>DetectStep: Write skip message to GITHUB_STEP_SUMMARY
        DetectStep-->>GitHubActions: has_changes=false
        GitHubActions-->>Developer: Job completes quickly (no heavy steps)
    else Relevant Rust changes
        GitHubActions->>SetupRust: Setup Rust toolchain
        SetupRust-->>GitHubActions: Rust ready

        GitHubActions->>InstallCargoMutants: cargo binstall cargo-mutants
        InstallCargoMutants-->>GitHubActions: cargo-mutants installed

        alt root_files not empty
            GitHubActions->>RootMutantsRun: cargo mutants --timeout-multiplier 3 --output mutants.out --file per root file
            RootMutantsRun-->>GitHubActions: mutants.out/outcomes.json
        end

        alt wt_files not empty
            GitHubActions->>WTMutantsRun: cargo mutants --dir wireframe_testing --timeout-multiplier 3 --output mutants-wt.out --file per wt file
            WTMutantsRun-->>GitHubActions: mutants-wt.out/outcomes.json
        end

        par Upload reports
            GitHubActions->>UploadArtifacts: Upload mutants.out/ as mutation-report-root (if present)
            GitHubActions->>UploadArtifacts: Upload mutants-wt.out/ as mutation-report-wireframe-testing (if present)
        end

        GitHubActions->>PostSummary: Run post_results for each report dir
        PostSummary-->>GitHubActions: Markdown with counts and surviving mutants tables
        GitHubActions-->>Developer: Job summary with per-crate results
    end

Loading

File-Level Changes

Change	Details	Files
Document daily, change-detection–gated mutation-testing workflow with split root vs wireframe_testing runs and updated jq-based reporting in the ADR.	Update ADR date and describe daily cron schedule guarded by a 24-hour git log–based change-detection step using commit timestamps. Define which Rust paths count as code changes, explicitly excluding manifest-only changes and documenting manual workflow_dispatch as a bypass. Describe separate cargo-mutants invocations for the root crate and the non-workspace wireframe_testing crate with per-file --file scoping. Revise execution, artefact, and summary behavior, including per-crate artefacts and a post_results helper that uses corrected jq field names and paths. Refresh risks, limitations, and add a Resolved Decisions section capturing schedule, change-detection, scope filtering, and workspace split handling.	docs/adr-007-mutation-testing-with-cargo-mutants.md
Update the execution plan to match the new daily, change-detection–guarded workflow, including file-pattern–based scoping, crate splitting, and parametrised jq post-processing.	Change the described schedule from weekly to daily, adding a change-detection guard that scopes runs to changed files using repeated --file arguments. Add constraints and discoveries around using git log --since, path patterns for code changes, and the need for a separate --dir wireframe_testing invocation. Expand the implementation stages to include a detect step, gating of heavy steps on has_changes, split mutation runs, and separate artefact uploads. Clarify validation, observable behavior, and artefacts to include skip behavior on quiet days and per-crate mutation-report-root / mutation-report-wireframe-testing artefacts. Parameterise the jq snippets on a directory argument and document corrected field references for counts and survivor tables.	docs/execplans/mutation-testing.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In the change-detection shell snippet, the case patterns use src/*.rs, examples/*.rs, etc., which won’t match files in nested directories (e.g. src/foo/mod.rs); consider switching to patterns that reflect the documented **/*.rs semantics or clarifying that only top-level files are considered.
• The ADR/execplan text says manual workflow_dispatch runs bypass change-detection and always run mutations, but the workflow sketch currently applies the has_changes guard unconditionally; it would help to either adjust the documented behaviour or show how the workflow will special-case manual runs (e.g. by setting has_changes=true for workflow_dispatch).

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In the change-detection shell snippet, the `case` patterns use `src/*.rs`, `examples/*.rs`, etc., which won’t match files in nested directories (e.g. `src/foo/mod.rs`); consider switching to patterns that reflect the documented `**/*.rs` semantics or clarifying that only top-level files are considered.
- The ADR/execplan text says manual `workflow_dispatch` runs bypass change-detection and always run mutations, but the workflow sketch currently applies the `has_changes` guard unconditionally; it would help to either adjust the documented behaviour or show how the workflow will special-case manual runs (e.g. by setting `has_changes=true` for `workflow_dispatch`).

## Individual Comments

### Comment 1
<location path="docs/execplans/mutation-testing.md" line_range="31" />
<code_context>
 To observe success: trigger the workflow manually via the GitHub Actions UI on
 the `main` branch, wait for it to complete, then check the job summary for the
-results table and download the artefact.
+results table (or skip message) and download the artefact(s).

 ## Constraints
</code_context>
<issue_to_address>
**nitpick (typo):** Consider simplifying "artefact(s)" to a straightforward plural for readability.

For consistency with the rest of the document, consider replacing `artefact(s)` with the simple plural `artefacts`.

```suggestion
results table (or skip message) and download the artefacts.
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6444281858

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/adr-007-mutation-testing-with-cargo-mutants.md`:
- Around line 177-205: The current change-detection block sets has_changes based
only on origin/main diffs (root_files, wt_files) but the workflow prose claims a
manual workflow_dispatch should bypass this guard; to fix, update the "Detect
changed Rust files" script to check the GitHub event (GITHUB_EVENT_NAME or
GITHUB_EVENT_NAME == "workflow_dispatch") at the top and, when a manual dispatch
is detected, set has_changes=true (and optionally set root_files/wt_files to a
sentinel like "*" or leave them empty) and write that to GITHUB_OUTPUT so the
subsequent guarded steps (the blocks that read has_changes) will run;
alternatively, if you prefer documentation change, update the prose to state
that manual dispatch does not bypass the has_changes guard—apply the chosen fix
consistently for the logic that reads has_changes, root_files, and wt_files.
- Around line 112-113: The manifest-only exclusion is inconsistent: the first
occurrence lists both Cargo.toml and Cargo.lock but the later section only
mentions Cargo.toml; update the ADR text so the excluded manifest set is
identical in both places by adding Cargo.lock to the later mention (the prose
that describes cargo-mutants filtering of Rust source files and the
manifest-only exclusion), ensuring both sections explicitly list "Cargo.toml"
and "Cargo.lock" so the definition is unified.

In `@docs/execplans/mutation-testing.md`:
- Around line 199-215: The plan currently contradicts itself about manual runs:
either manual (workflow_dispatch) runs should bypass the
`has_changes`/`root_files`/`wt_files` gates and always run mutation testing, or
they should obey the same gates; pick one authoritative behavior and make the
doc consistent. Update the numbered steps (2–9) and the acceptance/observable
behaviour text so they match that choice: if you choose bypass, add an explicit
clause and example `if:` semantics (e.g., "bypass when github.event_name ==
'workflow_dispatch' or when inputs.force == 'true'") and show how `has_changes`,
`root_files`, `wt_files` are ignored; if you choose to not bypass, remove the
manual-run claim (lines referencing manual runs always executing) and adjust the
`$GITHUB_STEP_SUMMARY`/`post_results` description to reflect gated execution.
Ensure all references to `has_changes`, `root_files`, `wt_files`,
`$GITHUB_STEP_SUMMARY`, and `post_results` are updated to the chosen semantics
so there is no contradiction.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b214fc8d-5a70-476e-a15e-dcea319e55dd

📥 Commits

Reviewing files that changed from the base of the PR and between 1380bd3 and 6444281.

📒 Files selected for processing (2)

• docs/adr-007-mutation-testing-with-cargo-mutants.md
• docs/execplans/mutation-testing.md

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/execplans/mutation-testing.md`:
- Around line 65-69: The doc contains contradictory policies about pinning
cargo-mutants (one place says "pin now" and another says "defer pin until after
first successful run"); pick one policy and make both occurrences identical.
Locate the two paragraphs that mention cargo-mutants, outcomes.json, jq filters,
and the mitigation (the Risk/Mitigation text and the later policy paragraph) and
update the wording so they state the same decision (either "pin the
cargo-mutants version in the workflow" or "defer pin until after first
successful run"), including a matching rationale and any action item (e.g., add
a comment noting the validated version if pinned). Ensure the text strings are
verbatim between the two sections so there is no contradiction.
- Around line 355-357: The three long jq command lines assigning to variables
caught, missed, and timeout exceed the project's 120-column Markdown wrap limit;
break each jq invocation into a multiline, wrapped form so the code block stays
<=120 columns while preserving the same jq filter and file reference (e.g., keep
the variable names caught, missed, timeout and the filter [.outcomes[] |
select(.scenario != "Baseline" and .summary == "...")] | length and the
"$dir/outcomes.json" input), ensuring line breaks occur at safe points within
the jq expression (after pipes or logical operators) to maintain readability and
identical behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1729cbfa-712a-43f3-bcaf-a48ece9ae46a

📥 Commits

Reviewing files that changed from the base of the PR and between 6444281 and dc68464.

📒 Files selected for processing (2)

• docs/adr-007-mutation-testing-with-cargo-mutants.md
• docs/execplans/mutation-testing.md