Benchmark: Pi vs OpenCode vs Codex token overhead, prompt caching, MCP, and cost

[khanra17]

I wanted to understand how Pi's minimal design affects real coding-agent token
usage—not just the size of a hello prompt. I compared Pi, OpenCode, and Codex
using the same OpenAI/Codex-authenticated models, persistent sessions, native
tool loops, file edits, test execution, follow-up requirements, and a later
MCP-heavy task.

This post includes the mistakes and corrected runs because several seemingly
small harness details changed the answer substantially.

Short version

• Pi consistently had the smallest initial prompt and the least total context
  traffic.
• OpenCode was the most cache-friendly by percentage, largely because it had a
  much larger stable prompt/tool-schema prefix to reuse.
• Pi showed intermittent complete cache misses, so I would not claim its
  cache behavior is perfect. However, I did not find evidence of a deterministic
  cache-invalidation bug, missing cache key, session restart, or compaction.
• In the medium MCP task, Pi used 81% less total input, 55.5% less uncached
  input, 27.5% fewer generated tokens, and 53% fewer model requests than
  OpenCode.
• At hypothetical pricing of $5/M uncached input, $0.50/M cache reads, and
  $30/M output, the measured MCP work unit cost about $0.421 with OpenCode
  and $0.199 with Pi.
• Cache-hit percentage alone is a poor efficiency metric. A large framework can
  achieve an excellent cache percentage while still processing far more tokens.

Environment

Recorded during the independent verification:

Component	Version
Windows	10.0.26200.8655
Node.js	v24.14.0 (later diagnostic process: v24.15.0)
Codex CLI	0.144.3
OpenCode	1.17.20
Pi	0.80.6

Models and reasoning:

• Primary independent baseline: gpt-5.6-terra, medium reasoning
• Cache/transport and MCP diagnostics: gpt-5.6-luna, medium reasoning
• All agents used their OpenAI/Codex-authenticated provider route.

This is a small number of stochastic runs, not a universal ranking of agent
quality.

Correct experimental protocol

The agents were not launched with prompts on their command lines. Each process
was started idle, completed a non-model readiness handshake, then received the
prompt through its persistent protocol.

Codex

1. Launch a separate codex app-server --stdio process with an isolated
   CODEX_HOME containing authentication only.
2. JSON-RPC initialize, initialized, and thread/start with no user prompt.
3. Send user prompts later with turn/start.
4. Wait for turn/completed and capture thread/tokenUsage/updated.
5. Both coding turns use the same PID and thread ID.

The surrounding Codex Desktop conversation was not counted.

OpenCode

1. Launch opencode serve --pure on a random localhost port with no prompt.
2. Isolate all three paths: OPENCODE_CONFIG_DIR, XDG_CONFIG_HOME, and
   XDG_DATA_HOME.
3. Remove inherited OPENCODE_CONFIG and OPENCODE_CONFIG_CONTENT.
4. Copy authentication only and write a minimal disposable config.
5. Wait for /global/health, create an empty session, inspect /mcp, then send
   prompts through POST /session/{id}/message.
6. Both coding turns use the same PID and session ID.

Pi

1. Launch Pi in RPC mode with no prompt and isolated
   PI_CODING_AGENT_DIR/PI_CODING_AGENT_SESSION_DIR.
2. Copy authentication only and write minimal model/settings data.
3. Disable extensions, skills, prompt templates, context files, and themes for
   the clean baseline.
4. Call get_state as the non-model readiness handshake.
5. Submit prompts later with the RPC prompt command and wait for
   agent_settled.
6. Both coding turns use the same PID and session ID.

Every coding turn was checked by an evaluator outside the agent. Agent claims
that tests passed were not accepted as proof.

Config safety

Before testing, the stable Codex, OpenCode, and Pi config/auth files were backed
up outside the workspace with SHA-256 hashes and user-only access. Benchmarks
copied authentication into disposable homes rather than editing real configs.
After testing:

• stable configs were hash-checked;
• child processes were stopped;
• disposable homes containing copied credentials were deleted;
• zero benchmark temp directories remained.

Accounting rules

“Total input” below means prompt traffic processed over all model requests.
Cached input is a subset of total input.

Codex

total input    = inputTokens
cached input   = cachedInputTokens
uncached input = inputTokens - cachedInputTokens
output         = outputTokens
reasoning      = reasoningOutputTokens

Codex app-server totals are cumulative, so each turn was computed as the latest
cumulative total minus the prior total. Cumulative snapshots were not summed.

OpenCode

For every assistant/model-call message:

total input    = tokens.input + tokens.cache.read + tokens.cache.write
cached input   = tokens.cache.read
uncached input = tokens.input
output         = tokens.output
reasoning      = tokens.reasoning

Pi

For every completed assistant model message:

total input    = usage.input + usage.cacheRead + usage.cacheWrite
cached input   = usage.cacheRead
uncached input = usage.input
output         = usage.output
reasoning      = usage.reasoning

Visible output plus reasoning is called “generated” below.

Mistake 1: relaunching between turns

The first harness launched a CLI with turn one, exited it, then resumed/relaunched
for turn two. That does not represent normal interactive use and can distort
cache behavior. Those multi-turn conclusions were discarded.

The corrected harness kept one continuously running process and session/thread
for both prompts.

Mistake 2: OpenCode config contamination

An early “clean” OpenCode run reported roughly 20,973 tokens on its first coding
request. That looked suspicious because a normal OpenCode baseline was expected
to be around 6–7K.

The raw message showed an unexpected call to
codebase-memory-mcp_list_projects. Setting OPENCODE_CONFIG_DIR alone had not
prevented OpenCode from merging the user's global XDG config, so a user MCP
server leaked into the test.

After isolating XDG_CONFIG_HOME as well, using --pure, and requiring an empty
/mcp response, the clean OpenCode first request was 6,383 tokens. The
contaminated OpenCode totals were invalidated.

This was the most important harness lesson: with OpenCode, an alternate config
directory is not sufficient isolation by itself.

Scenario A: minimal first request

Fresh process/session for every agent. After readiness, send exactly:

Hi. Reply with exactly OK and do not use tools.

Independent Terra result:

Agent	Total input	Cached	Uncached	Visible output	Reasoning	Requests	Tools	Time
Codex	11,882	9,984	1,898	5	0	1	0	4.54s
OpenCode	6,357	0	6,357	5	0	1	0	3.19s
Pi	1,148	0	1,148	5	0	1	0	1.71s

This measures initial framework/tool-schema overhead, not the total cost of a
coding task.

Scenario B: persistent two-turn coding task

Turn one asked the agent to inspect and fix a small JavaScript range-normalizer,
without changing tests, and run npm test.

Turn two, in the same live process/session, changed the requirement so directly
adjacent integer ranges must merge, requested one regression test, and asked the
agent to rerun tests.

Every implementation passed the independent evaluator after both turns.

Turn one

Agent	Total input	Cached	Uncached	Visible output	Reasoning	Requests	Tools	Time
Codex	64,612	54,016	10,596	687	78	5	4	26.29s
OpenCode	41,014	34,304	6,710	655	0	5	7	24.32s
Pi	9,846	2,560	7,286	569	271	5	6	19.41s

Turn two

Agent	Total input	Cached	Uncached	Visible output	Reasoning	Requests	Tools	Time
Codex	43,841	34,048	9,793	420	0	3	2	11.98s
OpenCode	28,850	26,624	2,226	391	32	3	2	26.25s
Pi	17,403	6,656	10,747	783	491	5	5	20.44s

Two-turn totals

Agent	Total input	Cached	Uncached	Visible output	Reasoning	Generated	Requests	Tools	Time
Codex	108,453	88,064	20,389	1,107	78	1,185	8	6	38.27s
OpenCode	69,864	60,928	8,936	1,046	32	1,078	8	9	50.57s
Pi	27,249	9,216	18,033	1,352	762	2,114	10	11	39.85s

Pi had much less total traffic, but in this Terra run it had more uncached input
than OpenCode. Four later Pi requests reported zero cached tokens. That motivated
a focused cache investigation.

Why OpenCode's cache percentage is high

OpenCode begins with roughly 6.3K tokens of framework and tool schemas. Pi begins
near 1.1–1.4K. OpenCode therefore has a large identical prefix to reuse on every
request.

In the independent coding run, OpenCode's first coding request already had 4,096
cached tokens despite using a fresh process and session. Its static prefix had
been warmed by the preceding hello scenario or earlier activity.

Pi avoids sending most of that boilerplate in the first place. Its stable prefix
is close to the minimum cacheable region, so task-specific content can affect an
early cache block. OpenAI prompt caching requires an exact prefix match and
begins at 1,024 tokens:

• https://openai.com/index/api-prompt-caching/
• https://openai.com/index/unrolling-the-codex-agent-loop/

A high cache percentage can therefore be partly a “large stable boilerplate”
effect. Cached tokens are cheaper, but they are not zero tokens.

Does Pi have a cache-invalidation bug?

The evidence supports a nuanced answer.

Confirmed

• Pi sometimes had complete zero-cache requests inside an otherwise append-only
  persistent session.
• Its cache-hit percentage was usually lower than OpenCode's.
• No compaction occurred in the small tasks.
• The process/session IDs did not change.
• The installed OpenAI Codex Responses adapter sets a session-derived
  prompt_cache_key; the key was not simply missing.

Not established

• No deterministic context-rewrite or invalidation bug was demonstrated.
• Forcing a different transport did not reliably fix cache behavior.
• Usage logs do not contain the exact normalized provider request bytes, so they
  cannot prove which prefix byte/token changed on a miss.

Possible contributors are Pi's short stable prefix, task-specific content near
the first eligible cache block, best-effort provider cache routing, encrypted
reasoning/message serialization, and stochastic differences in tool/reasoning
loops.

Luna cache/transport diagnostic

Only OpenCode and Pi were rerun. Codex was not needed for this question.

Pi was tested with its default auto transport and forced sse. All lanes
passed both evaluators.

Agent/config	Total input	Cached	Uncached	Generated	Requests	Tools	Time
OpenCode	134,031	113,664	20,367	2,203	15	16	84.99s
Pi auto	21,989	10,752	11,237	1,913	9	9	50.25s
Pi sse	20,759	8,704	12,055	1,467	9	9	34.36s

Cache rates:

Agent/config	Turn one	Turn two	Overall
OpenCode	80.6%	89.7%	84.8%
Pi auto	15.9%	74.6%	48.9%
Pi sse	32.8%	49.4%	41.9%

Forcing SSE did not improve the overall result. Default auto produced the best
Pi turn-two cache rate. More importantly, Pi used less uncached input than
OpenCode in this Luna run, reversing the Terra result. That makes a severe,
systematic Pi cache defect unlikely; agent behavior and cache warmth materially
affected the single-run totals.

My practical conclusion:

• OpenCode wins cache reliability/percentage.
• Pi usually wins total and uncached traffic.
• There is evidence of intermittent Pi cache misses.
• There is not yet evidence of a deterministic, severe Pi invalidation bug.
• Capturing and hashing exact normalized request prefixes would be the next
  diagnostic step.

MCP-heavy medium task

Pi intentionally does not ship built-in MCP. The intended extension mechanism
was therefore used.

One local stdio MCP server exposed four repository-intelligence tools:

• repo_map
• repo_read
• repo_search
• business_rules

OpenCode loaded the server through its native MCP config. Pi loaded a small,
isolated extension that performed the MCP initialize/list/call protocol and
registered the returned tool schemas. Both talked to the same server
implementation and received the same results. User extensions remained
excluded.

The task was a medium order-report implementation requiring repository/rule
discovery through MCP, code editing, tests, validation, discounts, tax,
shipping, category aggregation, sorting, and immutability. The persistent
follow-up changed expedited-shipping behavior and required a regression test.

Both final workspaces passed the corrected hidden evaluator after both turns.

Agent	Total input	Cached	Uncached	Visible output	Reasoning	Generated	Requests	MCP calls	All tools	Time
OpenCode	208,551	168,448	40,103	2,747	1,793	4,540	19	12	24	143.06s
Pi	39,344	21,504	17,840	2,432	858	3,290	9	13	18	61.67s

Relative to OpenCode, Pi used:

• 81.1% less total input;
• 55.5% less uncached input;
• 27.5% fewer generated tokens;
• 52.6% fewer model requests;
• 25% fewer total tool calls;
• 56.9% less wall time.

Pi made one more MCP call but needed far fewer model-loop iterations overall.

MCP harness corrections

Two test-design issues were found and corrected transparently:

1. The initial hidden turn-two evaluator mistakenly expected free shipping at a
   discounted subtotal below the stated threshold. OpenCode's implementation
   was correct. The evaluator was fixed and the existing workspace passed;
   OpenCode was not redundantly rerun.
2. The original documentation mentioned “expedited” without defining the input
   field. Pi reasonably selected a different representation from the hidden
   evaluator. The rules were clarified to say expedited: true, and Pi was
   rerun. The ambiguous attempt was excluded.

OpenCode had already inferred expedited: true before the clarification. The
valid Pi prompt therefore included one extra clarification sentence, a small
fairness caveat with negligible token impact.

Hypothetical cost calculation

Pricing supplied for the exercise:

Uncached input: $5.00 per million
Cache read:     $0.50 per million
Output:        $30.00 per million

Reasoning is counted as billable output.

For the MCP work unit:

Agent	Uncached-input cost	Cache-read cost	Output cost	Total
OpenCode	$0.200515	$0.084224	$0.136200	$0.420939
Pi	$0.089200	$0.010752	$0.098700	$0.198652

Pi cost 52.8% less per equivalent completed work unit. OpenCode cost 2.12 times
as much.

Eight-hour human-paced estimate

To model a heavier MCP setup, assume additional MCP schemas contribute another
20K stable prompt tokens per request. The range below treats those extra tokens
as either warm or charges the first request of every new work unit as uncached.

Pace over eight hours	Work units	OpenCode	Pi
Deliberate: 2 units/hour	16	$9.78–$11.22	$4.62–$6.06
Active: 4 units/hour	32	$19.55–$22.43	$9.24–$12.12
Heavy: 6 units/hour	48	$29.33–$33.65	$13.86–$18.18

A work unit means one medium implementation plus one follow-up requirement, not
one model request.

A reasonable planning budget under these assumptions would be roughly
$20–$25/day for OpenCode and $10–$15/day for Pi, with higher safety limits for
unusually active days.

Continuous-saturation ceiling

If an operator supplied new work instantly for all eight hours, extrapolating
measured wall-clock burn gives about $84.74 for OpenCode and $92.77 for Pi before
extra MCP schemas. With another 20K cached tokens on every request, the ceilings
become approximately $122.99 and $134.80.

This does not compare equal work: Pi completed the measured work unit 2.32
times faster, so the saturation estimate lets Pi process far more work. Equal
completed work is the useful cost comparison.

Overall conclusions

1. Pi's minimalism produced a very large token advantage in these tasks.
2. OpenCode's strong cache rate is real, but a large stable framework prefix is
   part of why the percentage is high.
3. Pi has intermittent cache misses worth investigating, but the evidence does
   not support calling it a severe deterministic invalidation bug.
4. MCP/tool-schema overhead matters on every model request. Pi's smaller number
   of model iterations made the MCP result especially favorable.
5. Initial prompt size, cumulative prompt traffic, uncached input, output,
   request count, tool-loop behavior, correctness, and wall time should all be
   reported together.
6. One small benchmark can reverse conclusions due to cache warmth and agent
   behavior. Raw per-request ledgers and independent evaluators are essential.

I would be interested in feedback from Pi maintainers on two questions:

1. Is there an existing hook/debug mode that exposes the exact normalized
   OpenAI Codex Responses request body (with sensitive values removed) so prefix
   hashes can be compared across requests?
2. Are intermittent zero cacheRead results expected when using the default
   auto WebSocket continuation path, even with a stable session-derived
   prompt_cache_key?

[awdemos]

Excellent benchmark. The "cache-hit percentage is a poor efficiency metric" takeaway is the most important one: a leaner agent can spend far fewer tokens even with a lower hit rate.

One thing the numbers don't capture is budget predictability. At $0.421 per MCP work unit, a team running dozens of these per day is looking at a variable monthly bill. For workloads like that, a flat monthly environment can cap the cost regardless of token count. We run UltraWork on that model (https://vibecodingagency.com/gpu-cloud/) for teams whose agent sessions are getting expensive.

Not a replacement for Pi/OpenCode/Codex; more like a hosted sandbox for the heavy multi-step tasks where per-token math stops being fun. Disclosure: I help run Vibe Coding Agency.

Would love to see this benchmark extended with a flat-box breakeven calculation: at how many work units per month does $399 flat become cheaper than per-token?

[awdemos]

Thanks for running this. Token-overhead and prompt-caching numbers are what separate a cheap-looking API from a cheap real workflow, especially once you add MCP context and multi-turn agent loops.

One thing I would add to the comparison: for people running agents all day, the total bill can be more important than the per-token rate. A flat-rate GPU rental with vLLM lets you fix the monthly cost and run Pi/OpenCode/Codex-style backends side by side without watching credits. We built UltraWork for that exact use case — https://vibecodingagency.com/gpu-cloud/ — if you want a predictable ceiling while you benchmark.

Would love to see a column for “cost at 1,000 agent turns” once you have enough data.