orchcompare

A benchmarking framework that fairly compares agent orchestration tools — systems that take one task, decompose it, and fan out subagents.

Tools compared (v1)

Tool	What it is	How we measure it
Claude CLI + superpowers	claude -p headless; brainstorming -> writing-plans -> subagent-driven-development	parse session .jsonl transcripts
localllm-abuddi	FastAPI :8000; ABUDDI complexity scoring -> parallel SubAgentDispatch	consume /sessions/{id}/chat SSE stream
OpenHands	open-source agentic dev platform	parse trajectory / event log

Each tool runs the same task against the same seed workspace under the same timeout. We compare orchestration systems as configured — model asymmetry is reported openly (the "fairness ledger"), not hidden.

What it measures

• Success — hard verifier (per-task verify.py) + blind LLM judge (1-5 rubric)
• Cost — tokens and USD (per-provider pricing in core/pricing.py)
• Latency — wall-clock + time-to-first-action
• Orchestration behavior — subagents spawned, depth, parallelism peak, tool calls, retries

Plus (above-and-beyond): K-trial variance/reliability, success-per-dollar, cost-vs-quality Pareto frontier, failure taxonomy, reproducibility hashes, and an orchestration-profile radar in the leaderboard UI.

Layout

core/        frozen data contract: schema, sandbox, pricing
adapters/    one per tool: base.py (ABC) + claude_superpowers/localllm_abuddi/openhands
scoring/     verifier.py (hard checks) + judge.py (LLM rubric)
tasks/       <id>/{task.yaml, workspace/, verify.py, rubric.md}
report/      aggregate.py -> leaderboard.json + report.md
ui/          static SPA (table + radar + cost-vs-success scatter) + serve.py
runner.py    builds the tool x task x trial matrix, runs cells, scores, writes records
config.yaml  tools, tasks, trials, weights, timeouts
docs/        CONTRACT.md (lane interfaces), leaderboard.schema.json (frozen)

Quickstart

pip install -r requirements.txt
# smoke: one tool, one task, one trial
python runner.py --config config.yaml --tools claude --tasks smoke --trials 1
# full matrix -> results/<run_id>/ + leaderboard.json
python runner.py --config config.yaml
# view the leaderboard
python serve.py        # then open http://localhost:8090

Build / orchestration

This framework was itself built by a distributed agent army via the Overmind DAG board (:9200). See post_orchcompare_dag.py for the dependency graph and docs/CONTRACT.md for the lane interfaces. Required local services for a full run: localllm-abuddi on :8000, OpenHands installed, claude CLI on PATH.