Fix mutation bugs#103
Merged
Merged
Conversation
added 3 commits
May 18, 2026 17:27
…rim mutants to better account for memory
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR introduces a mutant trimming framework to reduce mutation testing workload. A new Python utility, Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/mutation-evosuite.sh`:
- Around line 499-505: The script currently always runs trim_mutants.py using
$PYTHON_EXECUTABLE (and similarly at lines 545-547), which makes the -s (skip
mutation) mode fail on machines without Python; guard these mutation-only steps
so they only run when not in skip mode: detect the existing skip/skip-mutation
flag used by the script (the option that implements -s) and wrap the
"$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log"
calls in an if block that runs them only when skip mode is false, and otherwise
skip them (and do not try to locate Python). Ensure you update both occurrences.
- Around line 561-564: The script computes mutation_score using
mutants_generated and mutants_killed which will cause a division-by-zero crash
when mutants_generated is 0; update the logic around
mutants_generated/mutants_killed in scripts/mutation-evosuite.sh to check if
mutants_generated is zero before calling bc—if zero, set mutation_score to a
safe default (e.g., "0.00" or "N/A") and skip the bc division, otherwise perform
the existing bc calculation and printf formatting; reference the variables
mutants_generated, mutants_killed and mutation_score when implementing the
conditional guard.
In `@scripts/mutation-randoop.sh`:
- Around line 622-628: The pre-mutation steps still invoke trim_mutants.py (and
the runner-conversion step) unconditionally, causing a hard Python dependency
even when skip mode (-s) is requested; wrap the calls that use PYTHON_EXECUTABLE
(the trim_mutants.py invocation and the subsequent "runner conversion" call
mentioned around the same section) in a conditional that checks the skip-mode
flag (the variable set when -s is passed) and only runs those Python-dependent
commands when skip mode is not enabled, otherwise skip them (and optionally
print a short info message); also ensure the PYTHON_EXECUTABLE lookup remains
where used so Python is not required when skip mode is active.
- Around line 687-690: The division-by-zero happens when mutants_generated (from
summary.csv) is zero; update the logic in scripts/mutation-randoop.sh around the
variables mutants_generated, mutants_killed and mutation_score to guard the
calculation: check if mutants_generated is empty or equals 0 before running the
bc division, and if so set mutation_score to a safe default like "0.00" (or
"N/A") instead of performing the division; only compute mutation_score using
bc/printf when mutants_generated is non-zero.
In `@scripts/program-config/joda-time-2.3/build-evosuite.xml`:
- Line 86: The build file uses an undefined property `${test.pattern}` in the
mutation.test target and an incorrect path for `excludeMutantsFile`; update the
Ant properties so `test.pattern` has a sensible default (e.g., define a
`<property name="test.pattern" value="**/*Test*.class"/>` or similar) or accept
it as a required property, and change the `excludeMutantsFile` attribute
(referenced in the task that currently sets
`excludeMutantsFile="${resultdir}/../exclude_mutants.txt"`) to point to
`${resultdir}/exclude_mutants.txt` so it matches `trim_mutants.py` and other
projects; ensure references to `mutation.test` and the `excludeMutantsFile`
attribute are updated accordingly.
In `@scripts/trim_mutants.py`:
- Around line 143-147: The verbose print block uses total_original in a division
which will raise ZeroDivisionError when no mutants were parsed; update the block
that prints Mutants to keep/exclude (using variables selected_mutants,
excluded_ids, total_original, verbose) to guard against total_original == 0 —
either skip computing/printing Reduction or compute reduction as 0.0 when
total_original is zero, ensuring no division occurs and the prints still show
counts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: de4c8f6f-257e-4445-b01c-dff20208ea92
📒 Files selected for processing (63)
scripts/mutation-evosuite.shscripts/mutation-randoop.shscripts/program-config/ClassViewer-5.0.5b/build-evosuite.xmlscripts/program-config/ClassViewer-5.0.5b/build-randoop.xmlscripts/program-config/JSAP-2.1/build-evosuite.xmlscripts/program-config/JSAP-2.1/build-randoop.xmlscripts/program-config/a4j-1.0b/build-evosuite.xmlscripts/program-config/a4j-1.0b/build-randoop.xmlscripts/program-config/asm-5.0.1/build-evosuite.xmlscripts/program-config/asm-5.0.1/build-randoop.xmlscripts/program-config/bcel-5.2/build-evosuite.xmlscripts/program-config/bcel-5.2/build-randoop.xmlscripts/program-config/commons-cli-1.2/build-evosuite.xmlscripts/program-config/commons-cli-1.2/build-randoop.xmlscripts/program-config/commons-codec-1.9/build-evosuite.xmlscripts/program-config/commons-codec-1.9/build-randoop.xmlscripts/program-config/commons-collections4-4.0/build-evosuite.xmlscripts/program-config/commons-collections4-4.0/build-randoop.xmlscripts/program-config/commons-compress-1.8/build-evosuite.xmlscripts/program-config/commons-compress-1.8/build-randoop.xmlscripts/program-config/commons-lang3-3.0/build-evosuite.xmlscripts/program-config/commons-lang3-3.0/build-randoop.xmlscripts/program-config/commons-math3-3.2/build-evosuite.xmlscripts/program-config/commons-math3-3.2/build-randoop.xmlscripts/program-config/commons-primitives-1.0/build-evosuite.xmlscripts/program-config/commons-primitives-1.0/build-randoop.xmlscripts/program-config/dcParseArgs-10.2008/build-evosuite.xmlscripts/program-config/dcParseArgs-10.2008/build-randoop.xmlscripts/program-config/easymock-3.2/build-evosuite.xmlscripts/program-config/easymock-3.2/build-randoop.xmlscripts/program-config/fixsuite-r48/build-evosuite.xmlscripts/program-config/fixsuite-r48/build-randoop.xmlscripts/program-config/guava-16.0.1/build-evosuite.xmlscripts/program-config/guava-16.0.1/build-randoop.xmlscripts/program-config/hamcrest-core-1.3/build-evosuite.xmlscripts/program-config/hamcrest-core-1.3/build-randoop.xmlscripts/program-config/javassist-3.19/build-evosuite.xmlscripts/program-config/javassist-3.19/build-randoop.xmlscripts/program-config/javax.mail-1.5.1/build-evosuite.xmlscripts/program-config/javax.mail-1.5.1/build-randoop.xmlscripts/program-config/jaxen-1.1.6/build-evosuite.xmlscripts/program-config/jaxen-1.1.6/build-randoop.xmlscripts/program-config/jcommander-1.35/build-evosuite.xmlscripts/program-config/jcommander-1.35/build-randoop.xmlscripts/program-config/jdom-1.0/build-evosuite.xmlscripts/program-config/jdom-1.0/build-randoop.xmlscripts/program-config/joda-time-2.3/build-evosuite.xmlscripts/program-config/joda-time-2.3/build-randoop.xmlscripts/program-config/jvc-1.1/build-evosuite.xmlscripts/program-config/jvc-1.1/build-randoop.xmlscripts/program-config/nekomud-r16/build-evosuite.xmlscripts/program-config/nekomud-r16/build-randoop.xmlscripts/program-config/pmd-core-5.2.2/build-evosuite.xmlscripts/program-config/pmd-core-5.2.2/build-randoop.xmlscripts/program-config/sat4j-core-2.3.5/build-evosuite.xmlscripts/program-config/sat4j-core-2.3.5/build-randoop.xmlscripts/program-config/shiro-core-1.2.3/build-evosuite.xmlscripts/program-config/shiro-core-1.2.3/build-randoop.xmlscripts/program-config/slf4j-api-1.7.12/build-evosuite.xmlscripts/program-config/slf4j-api-1.7.12/build-randoop.xmlscripts/program-config/tiny-sql-2.26/build-evosuite.xmlscripts/program-config/tiny-sql-2.26/build-randoop.xmlscripts/trim_mutants.py
Comment on lines
+499
to
+505
| PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null) | ||
| if [ -z "$PYTHON_EXECUTABLE" ]; then | ||
| echo "Error: Python is not installed." >&2 | ||
| exit 2 | ||
| fi | ||
| "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log" | ||
|
|
There was a problem hiding this comment.
-s mode still requires Python, so skip path can fail before coverage runs.
trim_mutants.py (and runner conversion) are mutation-only steps, but they run before the skip gate. That makes -s fail on machines without Python.
🐛 Proposed fix
- PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null)
- if [ -z "$PYTHON_EXECUTABLE" ]; then
- echo "Error: Python is not installed." >&2
- exit 2
- fi
- "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log"
+ if [[ "$SKIP_MUTATION" -eq 0 ]]; then
+ PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null)
+ if [ -z "$PYTHON_EXECUTABLE" ]; then
+ echo "Error: Python is required for mutation trimming/runner conversion." >&2
+ exit 2
+ fi
+ "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log"
+ fi
@@
- if [ "$SUBJECT_PROGRAM" == "jdom-1.0" ]; then
+ if [[ "$SKIP_MUTATION" -eq 0 && "$SUBJECT_PROGRAM" == "jdom-1.0" ]]; then
"$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/convert_test_runners.py "$TEST_DIRECTORY" --mode evosuite-to-randoop
fiAlso applies to: 545-547
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/mutation-evosuite.sh` around lines 499 - 505, The script currently
always runs trim_mutants.py using $PYTHON_EXECUTABLE (and similarly at lines
545-547), which makes the -s (skip mutation) mode fail on machines without
Python; guard these mutation-only steps so they only run when not in skip mode:
detect the existing skip/skip-mutation flag used by the script (the option that
implements -s) and wrap the "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py
"$RESULT_DIR/mutants.log" calls in an if block that runs them only when skip
mode is false, and otherwise skip them (and do not try to locate Python). Ensure
you update both occurrences.
Comment on lines
+561
to
+564
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | ||
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | ||
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | ||
| mutation_score=$(printf "%.2f" "$mutation_score") |
There was a problem hiding this comment.
Handle zero generated mutants before computing mutation score.
If mutants_generated is 0, bc exits on division by zero and aborts the run due to set -e.
🐛 Proposed fix
mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv)
mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv)
- mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc)
- mutation_score=$(printf "%.2f" "$mutation_score")
+ if [[ "$mutants_generated" =~ ^[0-9]+$ ]] && [[ "$mutants_generated" -gt 0 ]]; then
+ mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc)
+ mutation_score=$(printf "%.2f" "$mutation_score")
+ else
+ mutation_score="0.00"
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | |
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | |
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | |
| mutation_score=$(printf "%.2f" "$mutation_score") | |
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | |
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | |
| if [[ "$mutants_generated" =~ ^[0-9]+$ ]] && [[ "$mutants_generated" -gt 0 ]]; then | |
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | |
| mutation_score=$(printf "%.2f" "$mutation_score") | |
| else | |
| mutation_score="0.00" | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/mutation-evosuite.sh` around lines 561 - 564, The script computes
mutation_score using mutants_generated and mutants_killed which will cause a
division-by-zero crash when mutants_generated is 0; update the logic around
mutants_generated/mutants_killed in scripts/mutation-evosuite.sh to check if
mutants_generated is zero before calling bc—if zero, set mutation_score to a
safe default (e.g., "0.00" or "N/A") and skip the bc division, otherwise perform
the existing bc calculation and printf formatting; reference the variables
mutants_generated, mutants_killed and mutation_score when implementing the
conditional guard.
Comment on lines
+622
to
+628
| PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null) | ||
| if [ -z "$PYTHON_EXECUTABLE" ]; then | ||
| echo "Error: Python is not installed." >&2 | ||
| exit 2 | ||
| fi | ||
| "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log" | ||
|
|
There was a problem hiding this comment.
-s mode still hard-depends on Python in the pre-mutation path.
Skip mode should allow generation + coverage without mutation tooling, but trim_mutants.py/runner conversion still execute before the skip gate.
🐛 Proposed fix
- PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null)
- if [ -z "$PYTHON_EXECUTABLE" ]; then
- echo "Error: Python is not installed." >&2
- exit 2
- fi
- "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log"
+ if [[ "$SKIP_MUTATION" -eq 0 ]]; then
+ PYTHON_EXECUTABLE=$(command -v python3 2> /dev/null || command -v python 2> /dev/null)
+ if [ -z "$PYTHON_EXECUTABLE" ]; then
+ echo "Error: Python is required for mutation trimming/runner conversion." >&2
+ exit 2
+ fi
+ "$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/trim_mutants.py "$RESULT_DIR/mutants.log"
+ fi
@@
- if [ "$SUBJECT_PROGRAM" == "hamcrest-core-1.3" ]; then
+ if [[ "$SKIP_MUTATION" -eq 0 && "$SUBJECT_PROGRAM" == "hamcrest-core-1.3" ]]; then
"$PYTHON_EXECUTABLE" "$SCRIPT_DIR"/convert_test_runners.py "$TEST_DIRECTORY" --mode randoop-to-evosuite
fiAlso applies to: 671-673
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/mutation-randoop.sh` around lines 622 - 628, The pre-mutation steps
still invoke trim_mutants.py (and the runner-conversion step) unconditionally,
causing a hard Python dependency even when skip mode (-s) is requested; wrap the
calls that use PYTHON_EXECUTABLE (the trim_mutants.py invocation and the
subsequent "runner conversion" call mentioned around the same section) in a
conditional that checks the skip-mode flag (the variable set when -s is passed)
and only runs those Python-dependent commands when skip mode is not enabled,
otherwise skip them (and optionally print a short info message); also ensure the
PYTHON_EXECUTABLE lookup remains where used so Python is not required when skip
mode is active.
Comment on lines
+687
to
+690
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | ||
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | ||
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | ||
| mutation_score=$(printf "%.2f" "$mutation_score") |
There was a problem hiding this comment.
Guard mutation score calculation when generated mutants is zero.
A zero value in summary.csv causes division-by-zero and aborts the script under set -e.
🐛 Proposed fix
mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv)
mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv)
- mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc)
- mutation_score=$(printf "%.2f" "$mutation_score")
+ if [[ "$mutants_generated" =~ ^[0-9]+$ ]] && [[ "$mutants_generated" -gt 0 ]]; then
+ mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc)
+ mutation_score=$(printf "%.2f" "$mutation_score")
+ else
+ mutation_score="0.00"
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | |
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | |
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | |
| mutation_score=$(printf "%.2f" "$mutation_score") | |
| mutants_generated=$(awk -F, 'NR==2 {print $2}' "$RESULT_DIR"/summary.csv) | |
| mutants_killed=$(awk -F, 'NR==2 {print $4}' "$RESULT_DIR"/summary.csv) | |
| if [[ "$mutants_generated" =~ ^[0-9]+$ ]] && [[ "$mutants_generated" -gt 0 ]]; then | |
| mutation_score=$(echo "scale=4; $mutants_killed / $mutants_generated * 100" | bc) | |
| mutation_score=$(printf "%.2f" "$mutation_score") | |
| else | |
| mutation_score="0.00" | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/mutation-randoop.sh` around lines 687 - 690, The division-by-zero
happens when mutants_generated (from summary.csv) is zero; update the logic in
scripts/mutation-randoop.sh around the variables mutants_generated,
mutants_killed and mutation_score to guard the calculation: check if
mutants_generated is empty or equals 0 before running the bc division, and if so
set mutation_score to a safe default like "0.00" (or "N/A") instead of
performing the division; only compute mutation_score using bc/printf when
mutants_generated is non-zero.
Comment on lines
+143
to
+147
| if verbose: | ||
| print(f"\nMutants to keep: {len(selected_mutants)}") | ||
| print(f"Mutants to exclude: {len(excluded_ids)}") | ||
| reduction = (len(excluded_ids) / total_original) * 100 | ||
| print(f"Reduction: {reduction:.1f}%") |
There was a problem hiding this comment.
Guard verbose reduction against empty mutant sets.
When the input log is empty (or no lines parse), total_original is 0 and verbose mode crashes on division by zero.
🐛 Proposed fix
if verbose:
print(f"\nMutants to keep: {len(selected_mutants)}")
print(f"Mutants to exclude: {len(excluded_ids)}")
- reduction = (len(excluded_ids) / total_original) * 100
+ reduction = (len(excluded_ids) / total_original) * 100 if total_original else 0.0
print(f"Reduction: {reduction:.1f}%")
print(f"Exclude list written to: {output_file}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/trim_mutants.py` around lines 143 - 147, The verbose print block uses
total_original in a division which will raise ZeroDivisionError when no mutants
were parsed; update the block that prints Mutants to keep/exclude (using
variables selected_mutants, excluded_ids, total_original, verbose) to guard
against total_original == 0 — either skip computing/printing Reduction or
compute reduction as 0.0 when total_original is zero, ensuring no division
occurs and the prints still show counts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.