How do you know if a harness is any good?
When Odysseus came home after twenty years, disguised as a beggar, not one person recognised him. His dog Argos did, from across the yard, and then died. Argos is the one who could tell the real thing from the disguise.
That is what a benchmark is for.
Status: open problem. See Getting started — this project runs async, in issues and pull requests, with no meetings.
A harness benchmark does exist. If you are here because someone said there wasn't one, that is out of date as of June 2026.
Claw-SWE-Bench (arXiv:2606.12344) is a real benchmark for harnesses, not models. It has 350 tasks across 8 languages and an 80-task short version. Its trick is an adapter protocol: it fixes the prompt, the time budget, the workspace, the way patches are extracted and the scoring, so that completely different harnesses can be compared fairly.
What it found is the reason this whole page exists:
| What you change | How much the score moves |
|---|---|
| The harness | 27.4 points |
| The model | 29.4 points |
Changing the scaffolding around the model is almost as powerful as changing the model. And one harness went from 19.1% to 73.4% purely by improving how it was wired up — same model, same tasks, different plumbing.
Two more pieces of work worth knowing:
- arXiv:2607.04528 shows the harness changes what the agent believes mid-task, even with the task, environment and model held fixed. Its conclusion: "harness design is an experimental variable in agent evaluation, not an implementation detail."
- HarnessBridge makes the harness itself a learnable component that sits between the agent and the world.
So the field is not empty. The gaps are elsewhere, and they are big.
Claw-SWE-Bench is a paper. What does not exist is a continuously updated public scoreboard where the model is held constant and harnesses are ranked against each other.
Today's leaderboards — Terminal-Bench, SWE-bench Verified — rank submissions, which are a model and a harness together. That is why the same model gets different scores from different vendors and nobody can tell you which part did the work.
Every benchmark above scores one run of one task. None of them measures:
- Does it survive a crash? Kill the process mid-run. Does it resume, or start over, or silently do the work twice?
- What does it cost? Tokens and dollars per task solved, not just whether it solved it.
- Does it hold up unattended? Run it a hundred times on a schedule. Does quality drift?
- Is it safe? Does it stay inside its sandbox, and can you tell afterwards what it did?
- Does it tell the truth? When it says a check passed, did the check run?
That last one is not hypothetical. In building the reference loop for these projects, the agent reported a clean check on a page that had zero links, and reported 2,938 words for an article that had 1,743 because it was counting stylesheet text as prose. Both reports looked fine. A benchmark that only scores task completion cannot see any of this.
Researchers found that the top three submissions to Terminal-Bench 2 were cheating, and that this kind of cheating happens in the scaffolding, not the model — so it is invisible if you only look at the model.
A harness benchmark needs anti-cheating built in from day one: held-out tasks, checks that the agent actually did the work rather than pattern-matching the test, and a way for anyone to re-run a submission.
If you want to compare two harnesses fairly, you have to fix a long list of things: the model and its version, temperature, the tool set, the time limit, the token budget, the starting files, retries, and what counts as done. Claw-SWE-Bench does this for coding tasks. There is no general write-up of how to run a fair harness comparison that someone could follow for a different kind of task.
Do not start by building a benchmark. Start by making the existing one easy to run.
- A runner. One command that takes a harness, runs Claw-SWE-Bench Lite (80 tasks) against it, and prints a score with the settings it used. If this is hard, nobody will ever compare anything.
- Adapters for four harnesses so the runner works with more than one thing: OpenClaw, Pi, Hermes, Codex — and Odysseus, the small harness from this workshop, as the honest baseline that should score lowest.
- A scoreboard that publishes results with the model pinned. Every row shows the exact settings. Anyone can re-run a row.
- New measurements the existing benchmarks skip — start with two, because they are the
easiest to do well:
- Cost per solved task, not just pass rate.
- Crash recovery: kill the process at a random point, restart, and check whether the work completed once, twice, or not at all.
- A fairness checklist. The written list of everything you must hold constant, so two people running the same comparison get comparable numbers.
Benchmarks are expensive and they rot. Running 350 tasks across several harnesses and several models costs real money in API credits, and the moment a harness updates, your numbers are stale. Plenty of benchmarks get published once and are never maintained. One person who looked at this problem said plainly that the API credits are the limiting factor.
The answer is to be small and honest about it. Start with the 80-task short version. Publish the cost of every run alongside the result, so people can judge how much to trust it. Pin harness versions in every row. If we cannot afford to keep it current, say so on the scoreboard rather than leaving stale numbers looking fresh.
And measure the boring things nobody else measures — cost and crash recovery — because those stay useful even when the task list goes out of date.
You do not need permission and you do not need to talk to anyone first.
- Read this README and HOW-WE-WORK.md. Five minutes.
- Pick an issue labelled
good first issue. Each one is written to be self-contained — enough detail to start without asking questions. - Comment "I'm taking this." That is the whole claiming process. If someone claimed it and has gone quiet for two weeks, take it anyway.
- Fork, branch, open a small pull request. One idea per PR. A 40-line PR gets read the same day; a 1,000-line one waits for a weekend.
- Add yourself to CONTRIBUTORS.md in that first PR — your name and what you are working on.
Have a bigger idea instead? Open a pull request adding rfcs/NNNN-your-idea.md using
the template, and argue for it in the PR. That is how anything
structural gets decided here.
Not sure where you fit? Open an issue that says what you know how to do and what you would enjoy. Someone will point you at the part of this that needs you.
- Run Claw-SWE-Bench Lite against any one harness and report what it cost you in dollars and hours. Nobody has published that, and everything else depends on it.
Apache 2.0.