fix: default rollout max_retries to 3

[mikasenghaas]

Summary

• Default the eval rollout retry count for transient infrastructure errors to 3 instead of 0.
• Updated EvalConfig.max_retries, the vf-eval --max-retries CLI flag (default + help text), and the eval config ingestion fallback.
• Updated the EvalConfig reference docs to match.

Breaking

• vf-eval / eval configs now retry transient infra errors up to 3 times by default. To restore the previous behavior, pass --max-retries 0 (or set max_retries = 0 in the eval config).

Made with Cursor

---

Note

Low Risk
Default-only behavior change for eval retries; no auth or data paths. Slightly longer runs on flaky infra unless callers opt out with max_retries=0.

Overview
Changes the default rollout retry count for transient infrastructure errors from 0 to 3 across eval configuration.

EvalConfig.max_retries, vf-eval --max-retries (default and help text), and the eval config builder fallback when max_retries is omitted now use 3. Reference docs for EvalConfig match.

Breaking: Evals retry failed rollouts up to three times by default (via maybe_retry around rollout/group execution). Use --max-retries 0 or max_retries = 0 in config to keep the old no-retry behavior.

^{Reviewed by Cursor Bugbot for commit 4cbdeba. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Default max_retries to 3 in rollout eval configuration

Changes the default value of max_retries from 0 to 3 in EvalConfig, the CLI argument parser, and the TOML config loader in eval.py. Documentation in docs/reference.md is updated to match. Behavioral Change: any eval run that previously relied on the default of 0 retries will now retry up to 3 times automatically.

^{Macroscope summarized 4cbdeba.}

[macroscopeapp]

Approvability

Verdict: Approved

Simple default value change from 0 to 3 for retry behavior on transient infrastructure errors. The change is consistent across all relevant locations (docs, CLI, config parsing, type definition) with clear intent and limited scope.

^{You can customize Macroscope's approvability policy. Learn more.}