Release 0.3.0

Release 0.3.0 - Claude 5.5 joins the frontier, Strands joins the harness list

October 2026

Version bump: MINOR -- adds four models to the omp /swe3 frontier file (Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, and Kimi K3 on Bedrock), and two Strands harnesses as the sixth and seventh supported agents. All of it is additive: no published score or cost from 0.2.0 moved.

---

Upgrading from 0.2.0

This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync.

git pull origin main
git checkout 0.3.0

cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

# Only for --agent strands or --agent strands-harness:
cd benchmarks && uv sync --group strands && cd ..

# Only for --agent omp against a Claude 5.5 model on Bedrock:
omp update   # 18.8.8 or later; see "Model resolution" below

Methodology changes

None. Numbers remain comparable with 0.2.0. Every model entry in the seven frontier files that existed at the 0.2.0 tag was compared with this release: not one existing model's score, cost per task or hosting group changed. pareto-frontier-omp-swe3.json gained claude-haiku-5-5, claude-opus-5-5, claude-sonnet-5-5 and kimi-k3-bedrock; the other six are unchanged. pareto-frontier-strands-swe3.json and pareto-frontier-strands-harness-swe3.json are new.

Two changes in this window read as methodology but are not:

• #210 groups a Bedrock model reached through the LiteLLM proxy (Path 2) as metered rather than self-hosted. The only model it affects, kimi-k3-bedrock, first appears in this release.
• #218 changes when the harness retries a task and fails a task whose replies came from the wrong model. Both apply to runs made from now on. No published run was re-scored.

Config changes

File	Change	Action needed
benchmarks/scripts/runner_config.py	strands and strands-harness added as agents; new api_key_env field names an environment variable that holds the endpoint key and wins over api_key	None. agent still defaults to claude, and api_key_env defaults to unset.
benchmarks/config/runner.example.yaml	Documents both Strands agents, their cost accounting and api_key_env	None.
benchmarks/pyproject.toml	New optional strands dependency group	uv sync --group strands before running either Strands agent.
benchmarks/scripts/bedrock_pricing.py	Regional and global rates for the Claude family (Haiku 4.5 through Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1) and for moonshotai.kimi-k3; the omp and pi agents fall back to this table when they report no cost	None. Adds rates, changes none.
self-hosted/vllm/pricing.json	Unchanged	None.

---

New models

All four run on omp /swe3 over mcp-gateway-registry-v2. Scores and costs come from each model's run-summary.json and pareto-frontier-omp-swe3.json; per-tier scores from models.json.

Model	Score	Cost per task	Scored	Frontier
claude-opus-5-5	81.16	$4.44	21/21	combined and Bedrock
claude-sonnet-5-5	79.49	$2.62	21/21	combined and Bedrock
kimi-k3-bedrock	72.87	$3.87	21/21	neither
claude-haiku-5-5	72.69	$0.14	20/21	combined and Bedrock

The combined frontier is now claude-haiku-5-5, qwen3.8-27b, claude-sonnet-5-5, claude-opus-5-5 and claude-opus-5. At 0.2.0 it was qwen3.6-35b, gemma-4-31b, qwen3.8-27b, kimi-k3, glm-5.3 and claude-opus-5. The Bedrock frontier moved from Haiku 4.5, Opus 4.5, Sonnet 5 and Opus 5 to the three Claude 5.5 models and Opus 5. The self-hosted frontier is unchanged.

Claude Opus 5.5

81.16 for $4.44 a task, 1.67 points behind claude-opus-5 (82.83) at 37% of its cost. Per tier it scores 81.52 trivial, 83.16 low, 79.97 medium and 80.24 high, so it holds its level on hard work. #195 ran it; #210 put it on the charts.

Its row was checked after this window found that omp fuzzy-matches model ids (see Bug fixes). A one-task re-run on omp 18.8.8, committed under claude-opus-5-5/omp/swe3/Hello-World, answered as Opus 5.5 on all 18 turns, and its omp-reported $1.1855 matches Opus 5.5 rates to the cent. #195's $93.27 matches the same rates; a fallback to Opus 5 would have cost $157 to $173.

Claude Sonnet 5.5

79.49 for $2.62 a task. It beats claude-sonnet-5 (76.97, $4.67) on score at 56% of the cost, and is the highest scorer on medium work (82.90). It finished all five hard tasks, at 73.24.

Claude Haiku 5.5

72.69 for $0.14 a task, the cheapest metered model on the chart: 16.51 points above claude-haiku-4-5 (56.18, $0.76) at under a fifth of the cost. Bedrock prices it at a tenth of Haiku 4.5 ($0.11 / $0.55 per million input / output tokens, regional), from the AWS price list. It falls away on hard work: 79.16 low, 60.00 high.

It completed 20 of 21 tasks. On registration-admission-control-gate it wrote all six artifacts to mcp-gateway-registry/ instead of the mcp-gateway-registry-v2/ path the prompt named 56 times, and its retry wrote nothing. That counts as a failure, as it would for any model; counted as 0, its mean over all 21 is 69.2. The misplaced files are kept under do-not-include/claude-haiku-5-5-misplaced-artifacts, outside every chart.

Kimi K3 on Bedrock

72.87 for $3.87 a task, the first published run on Path 2, an open-weight model on Bedrock reached through the LiteLLM proxy (#191). The same model self-hosted (kimi-k3) scores 79.17. Haiku 5.5 now matches it on score at 4% of its cost, so it sits on neither frontier.

---

New functionality

Strands Agents as a harness

--agent strands drives a task with an agent built on the Strands Agents SDK. Strands has no CLI, so the repo's strands_agent_runner.py is the agent process. It reports token counts but no billed cost, so the harness prices it from bedrock_pricing.py, as for codex. docs/strands-setup.md covers installing the optional group and the places the runner departs from Strands as shipped to finish long tasks on a vLLM endpoint. #194 added it; #204, #209 and #213 made it reliable.

--agent strands-harness runs the Strands team's own create_harness() agent through the same runner, with the same event stream, accounting and pricing (#212, docs/strands-harness.md).

Results over mcp-gateway-registry-v2, from harness-strands-swe3.md and harness-strands-harness-swe3.md:

Model	strands	strands-harness	omp, for comparison
claude-opus-5-5	81.21, $2.46	not run	81.16, $4.44
claude-sonnet-5-5	76.73, $0.96	77.81, $1.12	79.49, $2.62
claude-haiku-4-5	51.20, $0.70 (20/21)	56.05, $0.70	56.18, $0.76
minicpm5-2b	45.28 (21/21)	41.91 (20/21)	42.59 (19/21)

Opus 5.5 scores the same under Strands as under omp at 55% of the cost, because Strands caches the prompt more. The minicpm5-2b runs are self-hosted and priced from its throughput sweep on the results pages.

Benchmarking on your own infrastructure

#193 adds the onboard-benchmark skill for a first run against your own models, gateway and private repositories; a pre-flight check that every dataset repository is reachable at its pinned ref; api_key_env, so a real gateway token never sits in a config file; and workshop-template.yaml, a dataset template to copy. The template is a starting point, not a benchmark dataset.

---

What's changed

Skills

• New onboard-benchmark skill; the benchmark skill gains the repository pre-flight (#193).

Results and frontier

• omp /swe3 charts, frontier JSON and report regenerated for each new model (#210, #219).
• New results pages for both Strands harnesses (#207, #214, #215, #217).
• swe-router's models.json, allow-list, aliases and per-tier tables rebuilt for the Claude 5.5 frontier. The shipped allow-list is Opus 5, Opus 5.5, Sonnet 5.5, qwen3.8-27b and Haiku 5.5 (#219).
• The README badge now counts benchmarked models: 28 from 11 providers. The old "45 models" was a June count of models reachable on Bedrock (#219).

Documentation

• omp-setup.md gains a Model resolution section, ...

Release 0.2.0

Release 0.2.0 - Codex joins the harness list, MiniCPM5-2B joins the frontier

September 2026

Version bump: MINOR -- adds codex as a fifth supported coding harness, MiniCPM5-2B to the omp /swe3 frontier, and two benchmarked models that stay off it: Qwen3.6-35B-A3B FP8 on one L40S and Qwen3-Coder-30B-A3B on Bedrock. All of it is additive: no published score or cost from 0.1.0 moved.

---

Upgrading from 0.1.0

This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync.

git pull origin main
git checkout 0.2.0

# Sync the uv project(s) you run. No dependency changed in this release,
# so a sync is only needed if you are coming from a fresh clone:
cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

Methodology changes

None. Numbers remain comparable with 0.1.0. Every entry in the three frontier files was compared between the 0.1.0 tag and this release: pareto-frontier-omp-swe3.json gained minicpm5-2b, pareto-frontier-cc-swe3.json and pareto-frontier-combined-swe3.json gained kimi-k3, and not one existing model's score or cost per task changed.

The token-accounting fix in #184 reads as a methodology change but is not one: it only affects the codex agent, which had no published results before this release. The fleet-wide accounting fix that did move numbers shipped in 0.1.0.

Config changes

File	Change	Action needed
benchmarks/scripts/runner_config.py	codex added to VALID_AGENTS, with an is_codex property	None. agent still defaults to claude.
benchmarks/config/runner.example.yaml	The agent comment now names all five harnesses and how each one's cost is accounted	None.
self-hosted/vllm/pricing.json	Unchanged	None.
benchmarks/scripts/bedrock_pricing.py	Qwen3-Coder-30B-A3B priced at $0.1545 / $0.6180 per 1M input / output; cost_usd returns None instead of zero when a caller reports cached tokens against a model that cannot cache	None. Adds a rate, changes none.

---

New models

MiniCPM5-2B on omp / mcp-gateway-registry-v2

A 2.52B dense model scores 42.59 at $0.3993 per task, from pareto-frontier-omp-swe3.json. It completed 19 of 21 tasks; derive-repo-url-from-skill-md and registration-admission-control-gate produced no artifacts and are excluded from the mean rather than averaged in as zeros. Per tier it scores 43.76 trivial, 50.96 low, 40.36 medium, 33.45 high (models.json), so it holds up better on small changes than on architectural ones.

The serving guide is minicpm5-2b.md. Two things matter for anyone serving it: the weights are 5.03 GB in BF16 with a 128K native context, so it fits one L40S with room to spare, and it needs vLLM's minicpm5 tool parser because the model emits XML tool calls rather than JSON. The HuggingFace card still points at SGLang for tool calling, which is out of date.

Its throughput sweep (performance-summary.json) ran on g6e.4xlarge at $1.298/hr and peaked at 82.49 output tokens/sec at concurrency 7, with the cheapest blended rate of $0.04 per 1M tokens at concurrency 5.

Qwen3.6-35B-A3B FP8 on a single L40S, held off the frontier

FP8 on one L40S scores 59.23 under omp and 61.45 under codex over the same 21 tasks, against the BF16 four-GPU run's 59.24. Both runs live under do-not-include/qwen3.6-35b-fp8/ and reach no chart, because the model's only throughput sweep is on g6e.4xlarge rather than the canonical p5en.48xlarge arm the rest of the fleet is priced on, and its token counts are confounded with a 262144 context window against the BF16 run's 200000. The do-not-include README records both runs, their cost per task on the g6e basis ($0.5555 omp, $0.2645 codex at concurrency 10), and the token-accounting convention each harness uses.

The serving guide is qwen3.6-35b-a3b-fp8.md. The useful result: the model runs at its full 256K context on a $1.298/hr single-GPU box at the same quality as its BF16 parent.

Qwen3-Coder-30B-A3B on codex / Bedrock, off the frontier

The first benchmarked model on the codex harness against a hosted API rather than a local server. It scores 42.22 over 19 of 21 tasks at $0.6204 per task, $13.17 for the run, from its run-summary.json. Two model failures are excluded from the mean: cli-custom-egress-oauth-provider-flags filed no artifacts after 187 turns and 3.5M input tokens, and registration-admission-control-gate is missing review.md.

Reaching it needs a translating proxy, and that is a property of Bedrock, not of the harness. codex 0.153.4 speaks only the Responses API, and Bedrock serves Responses for the openai.* family alone: a Qwen request is rejected with The model 'qwen.qwen3-coder-30b-a3b-v1:0' does not support the '/openai/v1/responses' API. Of the two surfaces Bedrock does serve for Qwen, Converse is the one whose usage block can carry cache tokens, so the route is codex to a LiteLLM proxy to Converse. The committed litellm-mantle.yaml registers Qwen as openai/<id> against bedrock-mantle and forwards Responses untouched, so it cannot serve this path.

Every one of its 83,488,690 input tokens was billed fresh. cache_read is 0 on all 21 tasks, because Bedrock refuses a Converse cachePoint for the Qwen family with AccessDeniedException: You invoked an unsupported model or your request did not allow prompt caching, while the same request against claude-haiku-4-5 caches 5,204 tokens. That is what separates $13.17 metered here from the $5.56 a comparable codex run costs against self-hosted Qwen3.6-35B-FP8, where vLLM served 96.4% of the prompt from its prefix cache.

The run reaches no chart on two counts: the harness is codex, which no published frontier scans, and a metered Bedrock bill does not compare as raw dollars with the hardware-derived self-hosted points.

---

New functionality

codex as a fifth coding harness

--agent codex drives a benchmark run with OpenAI Codex (codex exec --json). Codex has no --skill flag, so the harness inlines the SKILL.md into the prompt, the way it already does for kiro-cli. It works on provider=bedrock through AWS_REGION and the ambient credential chain, and on provider=endpoint through a provider block carrying the base URL, with OPENAI_API_KEY in the environment. docs/codex-setup.md is the setup guide: which models each path can serve, the Responses-safe tool parsers a vLLM server needs, and the stdin redirect every codex run requires.

Codex reports token counts but no billed cost, so the harness derives cost from a new local price table, bedrock_pricing.py. A model missing from that table yields a null cost rather than a misleading zero.

Its token fields need their own convention. Codex reports input_tokens as the full prompt with the cached part as a subset, and the harness subtracts that out before recording, so the fields arrive mutually exclusive. token_accounting.py therefore declares codex's cache fields disjoint through DISJOINT_CACHE_AGENTS instead of detecting the shape from the data. Detection misfires when a codex run's cache hit rate lands near 50%: fresh input then equals cache_read + cache_write, the partition branch fires, and the total silently loses the whole cache read, halving the derived cost on the self-hosted path.

PR #162 added the agent. PR #184 made it runnable end to end and fixed the accounting.

---

What's changed

Documentation

• Root README and harness-reference.md now name all five supported harnesses. The agent reference had documented only claude and pi; it now covers omp, kiro and codex, including each one's cost basis.
• New codex-setup.md and the FAQ folder it feeds. omp and kiro-cli each had a per-harness setup page; codex's wiring lived scattered across the harness reference, the benchmark skill and a script docstring. The setup page installs codex, wires it to Bedrock and tabulates its failure modes; the FAQ's first entry, wiring codex to a model, carries the per-path recipes: the LiteLLM Responses-to-Converse bridge an open-weight Bedrock model needs, the Responses-safe tool parsers a vLLM server needs, and the direct route for openai.* models.
• New b300-cuda-fixes.md: the one-time driver and CUDA fixes a B300 node needs before vLLM will serve (#162).
• agent-cli-bedrock-setup.md gained the codex-on-Bedrock section: the native amazon-bedrock provi...

0.1.0 - First tagged release

Release 0.1.0 - First tagged release

September 2026

Version bump: first release. This tag establishes the versioning convention for the repository. From here, a new dataset or a new benchmarked model is a MINOR release, a change in methodology or completely new functionality is a MAJOR release, and everything else is a PATCH. See What each release will mean below.

---

What this repository is

This repository measures how well a language model does real agentic coding work, and what that work costs. It does not test a model in the abstract: it drives a model through a coding harness (the CLI agent that plans, edits files, and runs commands) over a dataset of real coding tasks, on one of three hosting paths, and scores the result with an independent judge.

So a result is always a triple: harness x model x dataset. Two benchmark types sit on top of that triple: a quality benchmark (can the model take a task from issue to a working patch, scored 0-100) and a throughput and cost benchmark (how much agentic load a self-hosted model sustains on a given GPU, and therefore what a task really costs).

Coding harnesses

A model is only as good as the harness driving it, so we benchmark several and keep every run.

We have settled on omp (oh-my-pi) as the headline harness. It drives the single-agent /swe3 skill, and the reported results are omp on the mcp-gateway-registry-v2 dataset. That is a deliberate choice, not the only one we support. The harness is a swappable part of the triple, and we run more than one on purpose so that a model's score can be read against the agent that produced it.

Harnesses with published runs in this release:

• omp (oh-my-pi) -- the headline harness, single-agent /swe3 on the v2 dataset. Setup and the three ways it differs from pi are in docs/omp-setup.md.
• Claude Code -- runs on both the single-agent /swe3 and the multi-agent /swe2 skills.
• pi -- runs on /swe3 and /swe2.
• kiro-cli -- landed as a third harness (#73), /swe3, 5 models, driving Kiro's managed Bedrock-backed models on a distinct Kiro-credit cost basis. See docs/kiro-cli-setup.md.

We will keep adding harnesses as they come up. opencode is next (#72). The benchmark is built so a new harness slots into the same triple without disturbing existing results, because scores are held per (harness, skill, dataset) and never merged across them.

Models

The headline omp / /swe3 run measures 18 models across the 21 tasks of the mcp-gateway-registry-v2 dataset. Every score and cost below is from docs/metrics/pareto-frontier-omp-swe3.json; the framing is from docs/results-index.md. Score is the mean judge score (0-100); cost is mean cost per task. The two cost bases do not compare as raw dollars -- a metered Bedrock bill against a hardware-derived self-hosted figure -- so read the models within each basis, per docs/cost-per-task-methodology.md.

On Amazon Bedrock (Anthropic family)

Model	Score	Cost/task
claude-opus-5	82.83	$11.95
claude-sonnet-5	76.97	$4.67
claude-opus-4-7	75.60	$7.35
claude-opus-4-8	74.69	$5.32
claude-opus-4-6-v1	70.64	$4.95
claude-opus-4-5	66.32	$4.18
claude-haiku-4-5	56.18	$0.76

Self-hosted open-weight (vLLM)

Model	Score	Cost/task
glm-5.3	80.11	$7.04
kimi-k3	79.17	$4.33
qwen3.8-27b	78.48	$1.47
kimi-k2.7-code	69.13	$2.47
deepseek-v3.2	61.66	$2.15
minimax-m3	60.51	$3.51
gemma-4-31b	59.74	$0.87
qwen3.6-35b	59.24	$0.26
minimax-m2.5	55.94	$0.48
devstral-2-123b	51.15	$0.79
qwen3-coder-30b	42.58	$0.47

The headline finding: claude-opus-5 tops quality at 82.83 for $11.95 a task, but the cheapest route into the high 70s is the open-weight qwen3.8-27b at 78.48 for $1.47 a task -- a 27B self-hosted model that outscores claude-sonnet-5 (76.97, $4.67) at under a third of the cost. At the low end, qwen3.6-35b scores 59.24 for 26 cents a task. Per-model serving guides (HF repo, context window, tensor-parallel size, tool-call parser, hardware fit) live in self-hosted/vllm/models/.

Datasets

Four datasets ship in benchmarks/dataset/. Scores from different task sets never merge, because the tasks, refs, and difficulty mixes differ.

• mcp-gateway-registry-v2 -- the headline dataset: 21 tasks across four complexity tiers. Each task is a closed issue from agentic-community/mcp-gateway-registry, pinned to the release before the fix shipped, so the defect is present in the tree the agent clones.
• mcp-gateway-registry (v1) -- the earlier 5-task set, kept as background.
• hello-world -- a smoke-test task for verifying a harness and hosting path work end to end.
• multi-repo-throughput -- the task set the throughput-and-cost sweep drives.

Hosting paths

A model can run in three places, all measurable by the same harness:

• Path 1 -- Anthropic models on Amazon Bedrock (published).
• Path 2 -- open-weight models on Amazon Bedrock through the LiteLLM proxy (works, not yet run).
• Path 3 -- self-hosted open-weight models on a vLLM server on EC2 (published).

What each release will mean

The "public interface" of this repository is its datasets, its measured models, and its measurement methodology. Versions follow Semantic Versioning, bare, with no v prefix:

• MAJOR -- a methodology change, or completely new functionality. A methodology change makes previously published numbers no longer comparable, which is the clearest reason to bump MAJOR.
• MINOR -- a new dataset, or a new model added to the frontier.
• PATCH -- everything else, backward compatible: doc and slide fixes, chart or frontier regeneration for an existing model, bug fixes, dependency bumps.

Releases are cut with the release-notes skill, which computes the bump from the change set, gathers the changes, writes the notes, and tags the merge commit through a PR.

Getting started

This repository is cloned and run, not deployed. There is nothing to upgrade from at 0.1.0:

git clone https://github.com/aarora79/agentic-coding-harness-benchmarks
cd agentic-coding-harness-benchmarks
git checkout 0.1.0

# Sync whichever uv project you run:
cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

On a fresh box, run the setup-machine skill first: it inspects the instance, names every missing dependency, and installs it. The guided tour from a checkout to a first run is docs/getting-started.md.

Contributors

Thank you to everyone who contributed to this release:

• Amit Arora (@aarora79)
• Prateek Sinha (@shekharprateek)

Media assets

Embedded media referenced by the README and the slide decks (not a software release). Currently: swe-router-explainer.mp4, the 70-second swe-router explainer.