feat: adds ability to optimize for cost

[andrewklatzke]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Describe the solution you've provided

Implements cost optimization in the same manner as latency optimization. Searches the acceptance statement for keywords pertaining to token usage/cost (e.g. costs, pricing, bill) and adds instructions to the variation generation to try to optimize for costs. Additionally has the acceptance statement prompt return instructions for the variation generation (ie, cheaper model, etc).

Describe alternatives you've considered

This is a feature addition.

Additional context

We'll be adding UI options for both latency and cost with adjustable thresholds, but these are still valid once those arrive since a mention of cost/latency means the user is trying to optimize for it.

---

Note

Medium Risk
Adds new cost-gating logic and changes iteration/batch bookkeeping (baseline tracking, history trimming, token-limit handling), which can affect optimization outcomes and persisted result records. Risk is moderated by extensive new unit tests covering the new gates and edge cases.

Overview
Adds cost optimization support alongside existing latency optimization: acceptance statements are scanned for cost keywords, agent calls get per-turn estimated_cost_usd (via model pricing when available), and a new _cost_gate is applied similarly to _latency_gate, with both gates recorded as synthetic judge scores for visibility.

Improves optimization loop correctness and observability by explicitly tracking baselines (duration and cost), trimming _history to bounded windows (standard and GT), counting variation-generation tokens into the run total, stamping accumulated_token_usage into result payloads, and refining token-limit behavior (treat 0 as unlimited and evaluate pass/fail before halting on budget). Also tightens model ID prefix stripping to avoid breaking Bedrock region-style IDs and updates package metadata naming/description.

^{Reviewed by Cursor Bugbot for commit 4fc1ecf. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Token count f-string may print None values

Low Severity

When agent_usage.input or agent_usage.output is None (which TokenUsage allows — estimate_cost explicitly guards against this), the f-string at this location would produce text like "used None input tokens and 40 output tokens" in the judge instructions. This happens because estimate_cost can return a non-None cost using only the non-None token count, but the f-string unconditionally formats both fields.

 

^{Reviewed by Cursor Bugbot for commit d267832. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default mode and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f2f0894. Configure here.}