Skip to content

Release 0.3.0

Choose a tag to compare

View all tags
@aarora79 aarora79 released this 11 Oct 14:32
· 6 commits to main since this release
0.3.0
1ec5326
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release 0.3.0 - Claude 5.5 joins the frontier, Strands joins the harness list

October 2026

Version bump: MINOR -- adds four models to the omp /swe3 frontier file (Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, and Kimi K3 on Bedrock), and two Strands harnesses as the sixth and seventh supported agents. All of it is additive: no published score or cost from 0.2.0 moved.

Upgrading from 0.2.0

This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync.

git pull origin main
git checkout 0.3.0

cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

# Only for --agent strands or --agent strands-harness:
cd benchmarks && uv sync --group strands && cd ..

# Only for --agent omp against a Claude 5.5 model on Bedrock:
omp update   # 18.8.8 or later; see "Model resolution" below

Methodology changes

None. Numbers remain comparable with 0.2.0. Every model entry in the seven frontier files that existed at the 0.2.0 tag was compared with this release: not one existing model's score, cost per task or hosting group changed. pareto-frontier-omp-swe3.json gained claude-haiku-5-5, claude-opus-5-5, claude-sonnet-5-5 and kimi-k3-bedrock; the other six are unchanged. pareto-frontier-strands-swe3.json and pareto-frontier-strands-harness-swe3.json are new.

Two changes in this window read as methodology but are not:

  • #210 groups a Bedrock model reached through the LiteLLM proxy (Path 2) as metered rather than self-hosted. The only model it affects, kimi-k3-bedrock, first appears in this release.
  • #218 changes when the harness retries a task and fails a task whose replies came from the wrong model. Both apply to runs made from now on. No published run was re-scored.

Config changes

File Change Action needed
benchmarks/scripts/runner_config.py strands and strands-harness added as agents; new api_key_env field names an environment variable that holds the endpoint key and wins over api_key None. agent still defaults to claude, and api_key_env defaults to unset.
benchmarks/config/runner.example.yaml Documents both Strands agents, their cost accounting and api_key_env None.
benchmarks/pyproject.toml New optional strands dependency group uv sync --group strands before running either Strands agent.
benchmarks/scripts/bedrock_pricing.py Regional and global rates for the Claude family (Haiku 4.5 through Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1) and for moonshotai.kimi-k3; the omp and pi agents fall back to this table when they report no cost None. Adds rates, changes none.
self-hosted/vllm/pricing.json Unchanged None.

New models

All four run on omp /swe3 over mcp-gateway-registry-v2. Scores and costs come from each model's run-summary.json and pareto-frontier-omp-swe3.json; per-tier scores from models.json.

Model Score Cost per task Scored Frontier
claude-opus-5-5 81.16 $4.44 21/21 combined and Bedrock
claude-sonnet-5-5 79.49 $2.62 21/21 combined and Bedrock
kimi-k3-bedrock 72.87 $3.87 21/21 neither
claude-haiku-5-5 72.69 $0.14 20/21 combined and Bedrock

The combined frontier is now claude-haiku-5-5, qwen3.8-27b, claude-sonnet-5-5, claude-opus-5-5 and claude-opus-5. At 0.2.0 it was qwen3.6-35b, gemma-4-31b, qwen3.8-27b, kimi-k3, glm-5.3 and claude-opus-5. The Bedrock frontier moved from Haiku 4.5, Opus 4.5, Sonnet 5 and Opus 5 to the three Claude 5.5 models and Opus 5. The self-hosted frontier is unchanged.

Claude Opus 5.5

81.16 for $4.44 a task, 1.67 points behind claude-opus-5 (82.83) at 37% of its cost. Per tier it scores 81.52 trivial, 83.16 low, 79.97 medium and 80.24 high, so it holds its level on hard work. #195 ran it; #210 put it on the charts.

Its row was checked after this window found that omp fuzzy-matches model ids (see Bug fixes). A one-task re-run on omp 18.8.8, committed under claude-opus-5-5/omp/swe3/Hello-World, answered as Opus 5.5 on all 18 turns, and its omp-reported $1.1855 matches Opus 5.5 rates to the cent. #195's $93.27 matches the same rates; a fallback to Opus 5 would have cost $157 to $173.

Claude Sonnet 5.5

79.49 for $2.62 a task. It beats claude-sonnet-5 (76.97, $4.67) on score at 56% of the cost, and is the highest scorer on medium work (82.90). It finished all five hard tasks, at 73.24.

Claude Haiku 5.5

72.69 for $0.14 a task, the cheapest metered model on the chart: 16.51 points above claude-haiku-4-5 (56.18, $0.76) at under a fifth of the cost. Bedrock prices it at a tenth of Haiku 4.5 ($0.11 / $0.55 per million input / output tokens, regional), from the AWS price list. It falls away on hard work: 79.16 low, 60.00 high.

It completed 20 of 21 tasks. On registration-admission-control-gate it wrote all six artifacts to mcp-gateway-registry/ instead of the mcp-gateway-registry-v2/ path the prompt named 56 times, and its retry wrote nothing. That counts as a failure, as it would for any model; counted as 0, its mean over all 21 is 69.2. The misplaced files are kept under do-not-include/claude-haiku-5-5-misplaced-artifacts, outside every chart.

Kimi K3 on Bedrock

72.87 for $3.87 a task, the first published run on Path 2, an open-weight model on Bedrock reached through the LiteLLM proxy (#191). The same model self-hosted (kimi-k3) scores 79.17. Haiku 5.5 now matches it on score at 4% of its cost, so it sits on neither frontier.

New functionality

Strands Agents as a harness

--agent strands drives a task with an agent built on the Strands Agents SDK. Strands has no CLI, so the repo's strands_agent_runner.py is the agent process. It reports token counts but no billed cost, so the harness prices it from bedrock_pricing.py, as for codex. docs/strands-setup.md covers installing the optional group and the places the runner departs from Strands as shipped to finish long tasks on a vLLM endpoint. #194 added it; #204, #209 and #213 made it reliable.

--agent strands-harness runs the Strands team's own create_harness() agent through the same runner, with the same event stream, accounting and pricing (#212, docs/strands-harness.md).

Results over mcp-gateway-registry-v2, from harness-strands-swe3.md and harness-strands-harness-swe3.md:

Model strands strands-harness omp, for comparison
claude-opus-5-5 81.21, $2.46 not run 81.16, $4.44
claude-sonnet-5-5 76.73, $0.96 77.81, $1.12 79.49, $2.62
claude-haiku-4-5 51.20, $0.70 (20/21) 56.05, $0.70 56.18, $0.76
minicpm5-2b 45.28 (21/21) 41.91 (20/21) 42.59 (19/21)

Opus 5.5 scores the same under Strands as under omp at 55% of the cost, because Strands caches the prompt more. The minicpm5-2b runs are self-hosted and priced from its throughput sweep on the results pages.

Benchmarking on your own infrastructure

#193 adds the onboard-benchmark skill for a first run against your own models, gateway and private repositories; a pre-flight check that every dataset repository is reachable at its pinned ref; api_key_env, so a real gateway token never sits in a config file; and workshop-template.yaml, a dataset template to copy. The template is a starting point, not a benchmark dataset.

What's changed

Skills

  • New onboard-benchmark skill; the benchmark skill gains the repository pre-flight (#193).

Results and frontier

  • omp /swe3 charts, frontier JSON and report regenerated for each new model (#210, #219).
  • New results pages for both Strands harnesses (#207, #214, #215, #217).
  • swe-router's models.json, allow-list, aliases and per-tier tables rebuilt for the Claude 5.5 frontier. The shipped allow-list is Opus 5, Opus 5.5, Sonnet 5.5, qwen3.8-27b and Haiku 5.5 (#219).
  • The README badge now counts benchmarked models: 28 from 11 providers. The old "45 models" was a June count of models reachable on Bedrock (#219).

Documentation

Bug fixes

  • omp ran a different model than the one requested. Its --model is a fuzzy match, so a Bedrock id missing from its catalog went silently to the nearest one. On omp 18.1.15 a 21-task "Haiku 5.5" run was Haiku 4.5 throughout; it was never published, and is kept as evidence under do-not-include. The harness now refuses an id missing from omp's catalog, warms that catalog in each task's agent directory (omp 18.8.8 still falls back from a cold one), and fails any task whose replies name another model as error_model_mismatch, recording models_used in metrics.json (#218).
  • A retry could delete finished work. omp can end a finished session on stopReason: toolUse; the harness retried, and the retry cleared all six artifacts before producing nothing. A task with every artifact on disk is no longer retried (#218).
  • The Strands runner on vLLM: compaction never fired below 200K, summarization sent an empty tools array, one large tool result could end a task, the cache-read fallback to Prometheus was missing, the skill loaded on demand rather than in the prompt, and it named tools the agent lacks (#204).
  • Results for a global. cross-region profile landed in a misnamed folder (#207).
  • The Strands event loop recursed per tool turn and crashed past about 320 turns; the limit is now 1,000 turns (#209), with a loop guard (#213).
  • Kimi K3 on Bedrock had no rates, and an omp or pi run that reported no cost recorded none; both now price from the table (#192).
  • A throughput-harness test failed intermittently on GitHub runners, whose coarse filesystem clock could stamp a new file before the level start; the stray-file sweep now allows 2 seconds of slack (#219).
  • vend/swe-router/models.json went stale on main when #219 was squash-merged, because it stamps the commit and date that last touched the frontier JSON. Restamped in this release. It is the same fault 0.2.0 fixed after #184.

Closed issues

Issue Title Closed by
#208 Strands harness: the event loop recurses per tool turn, so tasks past about 320 turns crash with RecursionError PR #209
#205 model_to_slug keeps the global. prefix, so Sonnet 5.5 results landed in a misnamed folder PR #207
#202 Strands harness: the swe3 skill names Claude Code tools the Strands agent does not have PR #204
#201 Strands harness: load the SWE skill into the prompt, as omp and kiro do, instead of on demand PR #204
#200 Strands harness records 0 cache-read tokens on vLLM instead of falling back to Prometheus PR #204
#199 Strands harness: one large tool result overflows the context window and ends the task PR #204
#198 Strands harness: context summarization fails on vLLM because OpenAIModel sends an empty tools array PR #204
#197 Strands harness ignores --context-window on an OpenAI-compatible endpoint, so proactive compaction never fires below 200K PR #204
#196 Benchmark minicpm5-2b under the Strands harness on mcp-gateway-registry-v2 (/swe3) PR #204

Pull requests included

PR Title
#219 benchmark(haiku-5-5, sonnet-5-5): Claude 5.5 on Bedrock via omp/swe3, v2 dataset
#218 Stop omp from running a different model than the one requested
#217 Add Sonnet 5.5 to the Strands harness results docs
#216 benchmark(sonnet-5-5): Sonnet 5.5 on Bedrock via strands-harness/swe3, v2 dataset
#215 Add a docs page for the Strands harness agent
#214 Add Strands harness results for Haiku 4.5 and MiniCPM5-2B on v2
#213 Add a loop guard to the Strands runner
#212 Add --agent strands-harness, the Strands team's create_harness() agent
#210 Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier, and count Path 2 runs as metered
#209 Raise the Strands runner's recursion limit to allow 1,000 tool turns
#207 Add the Strands results page and fix global. model slugs
#206 benchmark(haiku-4-5): Haiku 4.5 on Bedrock via strands/swe3, v2 dataset
#204 Benchmark minicpm5-2b under the Strands harness and fix Strands on vLLM
#203 benchmark(sonnet-5-5): Sonnet 5.5 on Bedrock via strands/swe3, v2 dataset
#195 benchmark(opus-5-5): Opus 5.5 on Bedrock via strands and omp, v2 dataset
#194 harness: add Strands Agents as a third harness option (--agent strands)
#193 Add private-repo pre-flight, env-sourced gateway key, and the onboard-benchmark skill
#192 fix(pricing): Kimi K3 Bedrock rates + cost fallback for omp/pi via LiteLLM
#191 benchmark(kimi-k3): Kimi K3 on Bedrock via omp/swe3, v2 dataset

Dependency updates

Package Previous Updated Project
strands-agents[openai] not used >=1.57.1,<2, optional strands group benchmarks
strands-harness not used >=0.1.2,<0.2, optional strands group benchmarks

Contributors

Thank you to everyone who contributed to this release:

Full Changelog: 0.2.0...0.3.0

Assets 2
Loading