0.1.0 - First tagged release

Release 0.1.0 - First tagged release

September 2026

Version bump: first release. This tag establishes the versioning convention for the repository. From here, a new dataset or a new benchmarked model is a MINOR release, a change in methodology or completely new functionality is a MAJOR release, and everything else is a PATCH. See What each release will mean below.

---

What this repository is

This repository measures how well a language model does real agentic coding work, and what that work costs. It does not test a model in the abstract: it drives a model through a coding harness (the CLI agent that plans, edits files, and runs commands) over a dataset of real coding tasks, on one of three hosting paths, and scores the result with an independent judge.

So a result is always a triple: harness x model x dataset. Two benchmark types sit on top of that triple: a quality benchmark (can the model take a task from issue to a working patch, scored 0-100) and a throughput and cost benchmark (how much agentic load a self-hosted model sustains on a given GPU, and therefore what a task really costs).

Coding harnesses

A model is only as good as the harness driving it, so we benchmark several and keep every run.

We have settled on omp (oh-my-pi) as the headline harness. It drives the single-agent /swe3 skill, and the reported results are omp on the mcp-gateway-registry-v2 dataset. That is a deliberate choice, not the only one we support. The harness is a swappable part of the triple, and we run more than one on purpose so that a model's score can be read against the agent that produced it.

Harnesses with published runs in this release:

• omp (oh-my-pi) -- the headline harness, single-agent /swe3 on the v2 dataset. Setup and the three ways it differs from pi are in docs/omp-setup.md.
• Claude Code -- runs on both the single-agent /swe3 and the multi-agent /swe2 skills.
• pi -- runs on /swe3 and /swe2.
• kiro-cli -- landed as a third harness (#73), /swe3, 5 models, driving Kiro's managed Bedrock-backed models on a distinct Kiro-credit cost basis. See docs/kiro-cli-setup.md.

We will keep adding harnesses as they come up. opencode is next (#72). The benchmark is built so a new harness slots into the same triple without disturbing existing results, because scores are held per (harness, skill, dataset) and never merged across them.

Models

The headline omp / /swe3 run measures 18 models across the 21 tasks of the mcp-gateway-registry-v2 dataset. Every score and cost below is from docs/metrics/pareto-frontier-omp-swe3.json; the framing is from docs/results-index.md. Score is the mean judge score (0-100); cost is mean cost per task. The two cost bases do not compare as raw dollars -- a metered Bedrock bill against a hardware-derived self-hosted figure -- so read the models within each basis, per docs/cost-per-task-methodology.md.

On Amazon Bedrock (Anthropic family)

Model	Score	Cost/task
claude-opus-5	82.83	$11.95
claude-sonnet-5	76.97	$4.67
claude-opus-4-7	75.60	$7.35
claude-opus-4-8	74.69	$5.32
claude-opus-4-6-v1	70.64	$4.95
claude-opus-4-5	66.32	$4.18
claude-haiku-4-5	56.18	$0.76

Self-hosted open-weight (vLLM)

Model	Score	Cost/task
glm-5.3	80.11	$7.04
kimi-k3	79.17	$4.33
qwen3.8-27b	78.48	$1.47
kimi-k2.7-code	69.13	$2.47
deepseek-v3.2	61.66	$2.15
minimax-m3	60.51	$3.51
gemma-4-31b	59.74	$0.87
qwen3.6-35b	59.24	$0.26
minimax-m2.5	55.94	$0.48
devstral-2-123b	51.15	$0.79
qwen3-coder-30b	42.58	$0.47

The headline finding: claude-opus-5 tops quality at 82.83 for $11.95 a task, but the cheapest route into the high 70s is the open-weight qwen3.8-27b at 78.48 for $1.47 a task -- a 27B self-hosted model that outscores claude-sonnet-5 (76.97, $4.67) at under a third of the cost. At the low end, qwen3.6-35b scores 59.24 for 26 cents a task. Per-model serving guides (HF repo, context window, tensor-parallel size, tool-call parser, hardware fit) live in self-hosted/vllm/models/.

Datasets

Four datasets ship in benchmarks/dataset/. Scores from different task sets never merge, because the tasks, refs, and difficulty mixes differ.

• mcp-gateway-registry-v2 -- the headline dataset: 21 tasks across four complexity tiers. Each task is a closed issue from agentic-community/mcp-gateway-registry, pinned to the release before the fix shipped, so the defect is present in the tree the agent clones.
• mcp-gateway-registry (v1) -- the earlier 5-task set, kept as background.
• hello-world -- a smoke-test task for verifying a harness and hosting path work end to end.
• multi-repo-throughput -- the task set the throughput-and-cost sweep drives.

Hosting paths

A model can run in three places, all measurable by the same harness:

• Path 1 -- Anthropic models on Amazon Bedrock (published).
• Path 2 -- open-weight models on Amazon Bedrock through the LiteLLM proxy (works, not yet run).
• Path 3 -- self-hosted open-weight models on a vLLM server on EC2 (published).

What each release will mean

The "public interface" of this repository is its datasets, its measured models, and its measurement methodology. Versions follow Semantic Versioning, bare, with no v prefix:

• MAJOR -- a methodology change, or completely new functionality. A methodology change makes previously published numbers no longer comparable, which is the clearest reason to bump MAJOR.
• MINOR -- a new dataset, or a new model added to the frontier.
• PATCH -- everything else, backward compatible: doc and slide fixes, chart or frontier regeneration for an existing model, bug fixes, dependency bumps.

Releases are cut with the release-notes skill, which computes the bump from the change set, gathers the changes, writes the notes, and tags the merge commit through a PR.

Getting started

This repository is cloned and run, not deployed. There is nothing to upgrade from at 0.1.0:

git clone https://github.com/aarora79/agentic-coding-harness-benchmarks
cd agentic-coding-harness-benchmarks
git checkout 0.1.0

# Sync whichever uv project you run:
cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

On a fresh box, run the setup-machine skill first: it inspects the instance, names every missing dependency, and installs it. The guided tour from a checkout to a first run is docs/getting-started.md.

Contributors

Thank you to everyone who contributed to this release:

• Amit Arora (@aarora79)
• Prateek Sinha (@shekharprateek)