BREAKING: Rename --max-attempts to --n-critic-runs

[juanmichelini]

Summary

This PR renames the --max-attempts parameter to --n-critic-runs across the benchmarks codebase to better reflect its purpose: controlling the number of critic evaluation runs in iterative mode.

Changes

• CLI argument: --max-attempts → --n-critic-runs
• Model field: max_attempts → n_critic_runs (EvalMetadata)
• Updated files: 17 files total
  • 7 run_infer.py files (all benchmarks)
  • Core logic files (evaluation.py, iterative.py)
  • 4 test files
  • 2 README documentation files

Breaking Changes

This is a breaking change for users. Existing scripts and workflows must be updated.

Migration Required

• CLI usage: --max-attempts 3 → --n-critic-runs 3
• Python API: EvalMetadata(max_attempts=3) → EvalMetadata(n_critic_runs=3)

Related PRs

This PR is part of a coordinated change. A corresponding PR will be created for the evaluation repo.

Testing

• All test files updated
• Changes verified across all benchmarks

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Pre-commit checks

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #325 at branch `rename-max-attempts-to-n-critic-runs`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[juanmichelini]

testing here https://github.com/OpenHands/evaluation/actions/runs/23027808880

[juanmichelini]

testing here https://github.com/OpenHands/evaluation/actions/runs/23027808880

[juanmichelini]

retesting here https://github.com/OpenHands/evaluation/actions/runs/23059734672

[juanmichelini]

@OpenHands bring all changes from main to this branch. resolve any conflicts like this:
main gets priority, then we make sure that we are respecting the "Rename --max-attempts to --n-critic-runs" even if we have to re do it.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

All changes have already been pushed to the remote branch rename-max-attempts-to-n-critic-runs. Here's the summary:

Summary

I merged all changes from main into the PR branch and re-applied the max_attempts → n_critic_runs rename, following the conflict resolution strategy: main gets priority, then re-apply the rename.

What was done

1. Merged origin/main (31 new commits) — no conflicts, auto-merged cleanly.
2. Re-applied the rename in 8 files that either came from main or were reverted by the merge:

File	Change
benchmarks/swebenchmultilingual/run_infer.py	New benchmark from main — renamed args.max_attempts → args.n_critic_runs, max_attempts= → n_critic_runs=
benchmarks/swefficiency/run_infer.py	New benchmark from main — same rename
tests/test_iterative_resume.py	max_attempts=3 → n_critic_runs=3 (2 occurrences reverted by merge)
tests/test_workspace_cleanup.py	max_attempts=1 → n_critic_runs=1 (3 occurrences reverted by merge)
benchmarks/commit0/config.py	Config dict key "max_attempts" → "n_critic_runs" + comment update
benchmarks/swtbench/README.md	--max-attempts → --n-critic-runs in CLI examples
benchmarks/utils/evaluation.py	Log string "max attempts" → "critic runs"
benchmarks/utils/models.py	Ruff auto-formatting fix

Checklist

• ✅ Main merged into branch (main gets priority)
• ✅ Rename re-applied to all new/reverted files
• ✅ No remaining max_attempts references (except unrelated retry logic in modal_patches.py)
• ✅ All pre-commit checks pass (ruff format, ruff lint, pycodestyle, pyright)
• ✅ All 157 tests pass
• ✅ Changes are concise — only the necessary renames, no extraneous modifications
• ✅ Pushed to origin/rename-max-attempts-to-n-critic-runs

[VascoSch92]

Just out of curiosity: why it is a breaking change? We don't have versions for benchmarks, right?

[juanmichelini]

@VascoSch92 I marked it as breaking because:

• it affects results considerably
• it needs to be merged with a sibling PR in evaluation

[juanmichelini]

@OpenHands a lot of time has passed, since this PR. Let's do this. match this branch with main, then rename again from scratch. That way we guarantee a clean merge

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

The request has been fully addressed:

• Reset branch to match main — Hard-reset rename-max-attempts-to-n-critic-runs to origin/main, giving a clean starting point
• Renamed from scratch — Re-applied max_attempts → n_critic_runs across 20 files (CLI arg, model field, core logic, all 9 benchmark run_infer.py files, config, 4 test files, 2 READMEs)
• Clean merge guaranteed — The branch now has a single commit on top of main with no merge history or conflict residue
• Pre-commit checks pass (ruff format, ruff lint, pycodestyle, pyright)
• Tests pass (10/10)
• Force-pushed to update PR BREAKING: Rename --max-attempts to --n-critic-runs #325

No extraneous changes — only the rename was applied. modal_patches.py was correctly left untouched since its max_attempts is unrelated retry logic, not critic runs.

[juanmichelini]

testing here https://github.com/OpenHands/evaluation/actions/runs/23303974373

[juanmichelini]

tested here https://openhands-eval-monitor.vercel.app/?run=swebench%2Flitellm_proxy-claude-sonnet-4-5-20250929%2F23303974373%2F