Benchmark v0.5.6

Benchmark: Agent Tasks on a Mid-Sized Unity 6 Project

Numbers come from valid runs only.

We gave the same four tasks to an AI agent four times over, once with each of four tools for controlling the Unity Editor. The project was a mid-sized Unity 6 URP project, not an empty sample.

This page shows the raw results and what we think they mean. As with the first benchmark, the goal is not to find "the best tool". Each tool makes different trade-offs. On most tasks the tools finished close together. They separated clearly on only one task. We hope this helps anyone choosing a setup for AI-driven Unity work.

Naming: unity-cli-bridge and the official Unity CLI are named. The two community tools are anonymized as Tool A and Tool B. This page compares design approaches, not specific projects. Its labels are independent of the first benchmark page, so "Tool B" here is not necessarily "Tool B" there.

Tools Tested

	unity-cli-bridge: Declarative CLI	Official Unity CLI: Registered Commands + Eval	Tool A: Dynamic Code Execution	Tool B: MCP Server
Design	Typed scene/prefab commands first (inspect, patch, add-component), with C# execute as a fallback	Editor-registered commands, batch, and eval	General-purpose C# execution is the core; other commands support that loop	Fine-grained MCP tools plus read-only MCP resources
Agent interface	Bash → CLI → JSON	Bash → unity command …	Bash → CLI → JSON	MCP tool calls (no shell needed)
Agent docs loaded	One plugin skill (small core + references on demand)	Official Claude plugin; agents loaded a ~50 KB CLI skill	Several project skills	MCP tool schemas
Version tested	v0.5.6 (measured at main 9f53c04, which includes PR #205, before the release was tagged). Older baseline: main at 04778d6	Unity CLI beta.13 + com.unity.pipeline 0.8.0-exp.1	Latest release as of 2026-10	Latest release as of 2026-10

Unity version: 6000.3.10f1 (Unity 6)
Platform: macOS arm64 (one Mac mini)
Project: A mid-sized Unity 6 URP project with an existing game codebase, nested prefabs and large serialized data
Agent: Claude Code with Opus 5.5 at --effort low, headless, one run at a time

Test Scenarios

Each scenario had a fixed English prompt. Each run started from a fresh copy of the project. A hidden verifier (EditMode/PlayMode tests the agent never sees) checked the result afterwards. A run passes only if every hard check passes, with no new compile errors.

ID	Scenario	What it stresses	Hard checks
s1c	Scene analysis & fix (C# allowed). Open an existing scene with a large prefab instance. Write a JSON audit report (grid size, per-type counts, missing references, broken props), then fix the planted problems. Edit grid cells by rule and add colliders sized to renderer bounds. Change only the scene instance, never the source prefab or assets.	Reading large serialized data and nested prefabs	7
s1q	Same task, commands only. The prompt adds: do not write or run C#, and use only the tool's query, inspect and edit commands.	How far each tool goes without code execution	7
s3	Pause-menu UI. Build a pause menu from the project's existing UI prefabs: Canvas setup, layout group, six button instances and a confirm popup. Write a small MonoBehaviour and wire each button with a persistent (serialized) onClick listener. Checked in Edit and Play Mode.	Prefab instancing, UI setup, a script recompile, Play Mode	7
s4	20 prefab variants. For 20 data assets, create a Prefab Variant of a base prefab and set a private serialized field. Sort the variants into folders, create 4 team materials and assign them, and label every variant. The base prefab and data assets must stay unchanged.	Batch asset work, serialized private fields	6

Results Summary

Mean of valid runs. Each cell is passed runs · mean wall time in seconds · mean cost in USD. Bold marks the best value in each row.

Scenario	Runs per tool	unity-cli-bridge	Official Unity CLI	Tool A	Tool B
s1c (C# allowed)	3	3/3 · 67.7 s · $0.307	3/3 · 63.7 s · $0.508	3/3 · 55.3 s · $0.320	3/3 · 69.3 s · $0.333
s1q (commands only)	5	5/5 · 119.9 s · $0.465	5/5 · 224.1 s · $0.897	4/5 · 454.9 s · $1.659	5/5 · 216.2 s · $1.977
s3 (pause-menu UI)	3	3/3 · 78.1 s · $0.284	3/3 · 99.1 s · $0.484	3/3 · 66.6 s · $0.218	3/3 · 80.0 s · $0.320
s4 (20 prefab variants)	3	3/3 · 32.7 s · $0.177	3/3 · 33.0 s · $0.344	3/3 · 34.9 s · $0.154	3/3 · 47.7 s · $0.244

Cost is the API-equivalent total_cost_usd that Claude Code reports. It is not an actual bill, but it tracks how much context each run used.
Wall time is the whole run: model thinking, tool calls, and Unity waiting on imports and recompiles.

Results by task type:

• C# allowed (s1c, s3, s4). All four tools passed every run. Every agent ended up doing the work through 1–5 C# snippets, whatever tool it had. Wall-time gaps here are mostly within run-to-run noise. Cost differences come mainly from fixed context size (see Fixed Context Cost).
• Commands only (s1q). This is the only task where the tools clearly separated. How much a tool's query and edit commands cover decided both whether the run passed and what it cost.

s1q Deep Dive: Commands Only

s1q is the same scene audit and fix as s1c, but the agent may not write or run C#. The verifier only fails a run if it leaves a new .cs file behind. Code-execution calls were counted separately, and no valid run used any (0 across all tools).

Metric (5 runs each)	unity-cli-bridge	Official Unity CLI	Tool A	Tool B
Pass	5/5	5/5	4/5	5/5
Wall time, mean	119.9 s	224.1 s	454.9 s	216.2 s
Wall time, median	84.7 s	232.8 s	250.8 s	181.6 s
Wall time, range	58.7–295.3 s	157.9–262.0 s	42.8–1,026.8 s	141.5–352.0 s
Cost, mean	$0.465	$0.897	$1.659	$1.977
Cost, range	$0.44–0.50	$0.79–1.01	$0.35–4.02	$1.60–2.28
Tool calls, mean	13	27	36	63
Output tokens, mean	5,626	8,746	19,932	19,081

How each agent approached it:

• unity-cli-bridge. Agents read the scene with scene inspect --omit-defaults, which includes the enabled/disabled state inside nested prefabs. They then fixed it with scene add-component and scene patch, using 11–18 calls. One run (r4, 295.3 s) took much longer than the others. Most of that was tool time, and we have not yet looked into why. That run pulls the mean well above the median.
• Official Unity CLI. Agents combined get_serialized_fields, get_component_properties, get_scene_hierarchy and batch, often glued together with shell functions. The results were consistent, but each command took about 750 ms, so tool time alone was 88–139 s per run.
• Tool A. It has no non-code command for editing a scene, so its agents improvised. One passing run parsed the scene YAML and model files with scripts and edited the YAML text directly. The failed run is a valid failure: the agent stopped and asked for permission, saying it had no way to change the scene without code execution. It stays in the averages. Two of the passing runs took 12–17 minutes.
• Tool B. The scene and component tools worked, but agents read MCP resources one object at a time, which meant many round-trips (59–69 calls). Results returned to the model came to roughly 160–220 KB per run where we measured it. That made Tool B the most expensive on this task, even though its wall time was mid-pack.

Key insight: Most of the gap comes from whether a tool's read output already holds the needed state (here, disabled renderers inside nested prefabs). When it doesn't, the agent falls back to digging through YAML, and both time and cost multiply. The next section shows the same effect within unity-cli-bridge.

Before / After: unity-cli-bridge PR #205

Our first s1q runs used unity-cli-bridge main at 04778d6. Those agents also fell back to grepping and parsing scene and prefab YAML. We then fixed what the transcripts showed, in PR #205:

• scene inspect --omit-defaults now keeps enabled state and nested prefab information instead of dropping it as a default.
• The scene patch spec ships inside the skill as a reference file, and the agent can open it when needed.
• scene patch suggests the closest valid operation when it gets an unknown one.
• prefab inspect accepts model files (.fbx).
• qa world-dump works in Edit Mode.

s1q (commands only)	Runs	Pass	Wall time, mean	Cost, mean	Tool calls, mean	Output tokens, mean
main 04778d6 (before)	2	2/2	332.8 s	$1.127	40	15,144
main 9f53c04 (after PR #205)	5	5/5	119.9 s	$0.465	13	5,626
Change			−64%	−59%	−68%	−63%

On s1c, s3 and s4 (C# allowed), before and after stayed within noise of each other: 67.7 vs 56.5 s, 78.1 vs 68.0 s, and 32.7 vs 32.5 s. Cost was slightly higher after, by $0.00–0.05 per run.

Caution: These fixes target failures we saw in this benchmark, and they were re-measured on the same task. Read the before/after as "the specific gap we saw is closed", not as a general speed-up. The other tools were measured once, as released, with no chance to tune against these tasks.

Fixed Context Cost

What each tool puts into the agent's context before any work starts. Tokens are from the first model turn.

	unity-cli-bridge	Official Unity CLI	Tool A	Tool B
First-turn context	~14.1k tokens	~17.1k tokens (+ ~50 KB skill loaded on first use)	~15.6k tokens	~16.4k tokens

On short C#-allowed tasks this baseline matters most. The official CLI's larger skill raised cache-creation tokens, and its cost per run was 1.4–2.3× the others on s1c and s4 ($0.508 and $0.344).

Per-Call Latency

These were measured with the earlier small-project setup, for code execution, the main operation all four tools share:

	unity-cli-bridge execute	Official eval	Tool A code execution
Latency	~0.5 s for a new snippet (compile); ~83 ms end-to-end when cached	~750 ms	~19 ms (Edit Mode), ~14 ms (Play Mode)

Tool A's execution is an order of magnitude faster than ours. On tasks that run many small snippets, such as driving Play Mode frame by frame, this matters more than anything measured above. In an earlier small 2D-template round (2 runs per tool), Tool A was the fastest and cheapest tool for automated Play Mode testing.

Design Trade-off Analysis

unity-cli-bridge: Declarative CLI

Strengths:

• Broadest typed scene and prefab command set in this test (inspect/patch specs, --omit-defaults, add-component, prefab inspect)
• Fastest and cheapest on the commands-only task (s1q: 5/5, 119.9 s mean, 84.7 s median, $0.465)
• Smallest fixed context (~14.1k tokens); agents opened reference docs only when they needed them
• Structured JSON on stdout, so no extra call is needed to read results

Weaknesses:

• execute still compiles each new snippet (~0.5 s), far slower than Tool A for repeated code execution
• The s1q advantage only appeared after PR #205; the older main fell back to YAML digging like the others
• One s1q run took 295 s, and we don't yet know why
• The editor package and CLI versions must match

Official Unity CLI: Registered Commands + Eval

Strengths:

• Made by Unity; one binary also covers editor install, licensing, project management and builds
• Passed s1q 5/5 using only registered commands and batch, with the most consistent times among the tools (157.9–262.0 s)
• Second-lowest cost and output tokens on s1q
• The plugin ships many domain skills (UI, URP, audio, …)

Weaknesses:

• ~750 ms per command, so tool time adds up
• Largest fixed context; on short tasks its cost was the highest (s1c $0.508, s4 $0.344)
• Requires Unity 6+, still an experimental beta; plugin and skills install globally by default

Tool A: Dynamic Code Execution

Strengths:

• Very fast code execution (~19 ms)
• Its skills teach good patterns for Play Mode automation and repeated checks
• Fastest on s1c and s3, and cheapest on s3 and s4

Weaknesses:

• No real way to edit a scene without code execution: s1q 4/5, slowest on average, and the widest spread (42.8–1,026.8 s)
• Setup friction in our harness: a package-setup dialog froze the editor once, and setup windows open with the editor

Tool B: MCP Server

Strengths:

• Standard MCP, so it connects to any MCP client without a shell
• Wide built-in coverage (scenes, components, assets and more)
• Passed every run in every scenario

Weaknesses:

• Reading a large scene through per-object resources caused many round-trips and large results (s1q: the most expensive at $1.977, with 59–69 calls)
• Needs a Python runtime alongside the editor

Methodology

• Same machine, same Unity version, same project snapshot, and same prompts for all tools. Each run got a fresh file-system clone of a pre-imported project.
• Runs were sequential, with a 2-minute gap and a 60-minute cap. A hidden verifier ran after each run, and the agent never saw it.
• Each tool was installed the way its docs describe, but scoped to the run. The agent got that tool's own skill or plugin and nothing else.
• Isolation. Agents were blocked, by permission rules and a pre-tool hook, from reading any tool's source code on disk. Only the current run's own skill docs were allowed, and file-system-wide searches were blocked.
• Compatibility shim. The project's game code referenced one unity-cli-bridge attribute. Runs for the other tools got a small stub so the project compiled the same way.
• Metrics came from Claude Code's stream output: wall time, total_cost_usd, output tokens, tool calls, and code-execution calls.
• Excluded records (not in any number above):
  • 2 early runs of the old unity-cli-bridge baseline. The agent read unity-cli-bridge's package source on disk before isolation was tightened. Both were re-run under the stricter isolation. Notably, reading the source had made those runs faster.
  • 1 Tool A run on s1c. A package-setup dialog blocked the editor's main thread (a harness problem). It was re-run after the harness was fixed.
  • 1 attempt that aborted before starting, because a previous editor was still open. It produced no data and was re-run.
  • All dry and trial runs.
  • Tool A's failed s1q run is not excluded. It was a valid run and counts as a failure.

Limitations

• Small samples. There were 5 runs per tool for s1q and 3 for the others (2 for the old unity-cli-bridge baseline). Spread within one tool is large, and a single slow run moves the mean, so we report medians and ranges for s1q. Treat differences of 10–20% as noise.
• One model, one client. We used only Claude Code with Opus 5.5 at low effort. Other models, effort levels or clients may rank the tools differently.
• One machine, one Unity version. Background indexing on the machine sometimes raised system load during runs.
• Run by the unity-cli-bridge author. We designed and ran the benchmark, and we fixed our own tool between rounds. The other tools were measured as released. We tried to be fair (fixed prompts, hidden verifier, source isolation), but keep this bias in mind.
• Narrow tasks. Four tasks on one project. In particular, s1q is one specific kind of scene work.

Conclusion

Each approach serves different needs:

• Scene and prefab edits without code (reviewable, command-level changes) → On this task, the declarative command approach had the edge: 5/5, about half the time and cost of the next tool. The official CLI also passed every run, at a steady pace.
• Fast, repeated code execution (Play Mode automation, runtime checks) → The dynamic-code approach (Tool A) is much faster per call and did best on those tasks.
• One-shot C# batch work → Any of the four is fine. Differences were mostly noise, and cost tracked fixed context size.
• Any MCP client, no shell → An MCP server (Tool B) connects most broadly, but reading large scenes object by object gets expensive.
• Official path for editor install, licensing, builds and CI → The official Unity CLI covers all of it, at the cost of per-call latency and a large skill.

Once code execution is allowed, all four tools look much alike. The real differences show up when it isn't, or when it has to run many times.

---

Benchmark conducted on 2026-10-20 (KST). Results may vary with other models, Unity versions, projects and hardware. Tool A and Tool B are anonymized: this comparison focuses on design approaches, not specific projects. Their labels are independent of the earlier benchmark page.