Independent benchmark of rtk on a pi/forge agentic workload — request for review before June 17 publication

[devasur]

Hello — I run an independent, reproducible benchmark of context-management middleware on a real agentic SDLC workload (forge-cli/4ge harness on the pi-coding-agent runtime, ollama-cloud models). rtk 0.42.2 is one of three products evaluated, via your pi extension (rtk init --agent pi).

Everything is public and reproducible: https://github.com/Entelligentsia/tokbench — Dockerfiles (your musl release binary vendored + sha256-verified), raw per-run transcripts, a frozen pre-registered protocol, and mechanism notes. Replication runs execute over the coming days; we plan to publish a story on June 17th and would value your review before then.

What we observed (pilot, N=1 — details in notes/02 and notes/03): rtk worked exactly as designed. 74 commands rewritten, 74.7% compression on the commands it touched, ~152ms average overhead, zero rtk-caused failures — its best moment was compressing a failing eslint run from ~54K tokens to a usable summary, with the agent still recovering and all quality gates green. The finding that may interest you is about addressable surface, not quality: on this harness, registry-covered bash output was only ~2.5% of total billed input — the harness's hottest uncovered paths were node <script>.cjs invocations (the harness's store CLI) and npm install, both passthrough.

Our asks: (a) is our setup faithful (project-local rtk init --agent pi, telemetry disabled, registry at defaults)? (b) any comment on registry coverage for node <script> patterns; (c) we will include a maintainer response verbatim in the publication if you'd like to provide one.

Disclosure: the benchmark harness (forge-cli) is my own product — scope and conflicts statement in the repo README. Findings are scoped to this harness/provider; the writeup explicitly states rtk performed as designed.

Thanks — the pi extension's rewrite-only design and the RTK_DISABLED toggle made rtk the easiest of the three products to integrate and test fairly.

[pszymkowiak]

wshm · Automated triage by AI

🔍 Automated Triage

📋 Category	question
🟡 Priority	medium
🎯 Confidence	87%

Summary

External researcher seeks maintainer review of independent rtk benchmark before June 17 publication; asks three specific questions about setup fidelity and registry coverage — no bug or feature request, needs a timely response.

---

Triaged automatically by wshm · This is an automated analysis, not a human review.

[devasur]

Benchmark results on a phase-isolated SDLC harness (forge-cli) — request for your read

We ran rtk 0.42.2 through a pre-registered benchmark (frozen protocol, every transcript public) on forge-cli, an 8-phase SDLC pipeline, on a request-metered provider (ollama-cloud), against a native (no-middleware) baseline. n=3 per arm. Before publishing (June 17) we'd like you to confirm or correct our characterization — because the headline number is not rtk's fault, and we want to say that precisely.

rtk worked exactly as designed

Engagement was clean every run: rtk rewrote 47–79 commands/run, saving 64–76% on the commands it touched (R04 54 cmds / 30.3K saved / 75.7%; R06 79 / 42.0K / 67.3%; R11 47 / 32.1K / 64.4%). The rewriting is invisible and automatic, as intended. No faults.

The result is an addressable-surface story, not a cost story

rtk's three runs billed 2,147,541 / 2,839,459 / 2,959,227 input tokens (median +30.1% vs native median 2,183,153). But that +30% is run-path variance, not rtk overhead — and we want to be clear about why:

• rtk is a PreToolUse:Bash command rewriter. It injects nothing into context, adds no tools, and changes no prompt — so it is mechanically incapable of increasing the token bill. It can only shrink command output.
• Its lowest run (2,147,541) lands at native (−1.6% vs native median) — i.e. rtk has no cost floor; the high runs are which phases happened to be heavy, not rtk.
• Its addressable surface here was ~1–2.5% of the bill, because forge agents spend their tokens on file reads and a structured store accessed via MCP tools (forge_store/forge_artifact) — neither is a Bash command in rtk's recipe table, so rtk never sees them. (forge's own per-call compression already cuts that store traffic ~72%, separately.)

So: rtk's savings are real but small because the surface is small on this harness, and the apparent +30% is noise we can't attribute to rtk. We read this as architecture–surface fit, not product quality.

What we'd like you to confirm or correct

1. Is our mechanism read right — that rtk only acts on Bash command invocations it has recipes for, and therefore cannot add tokens (so the +30% is variance, not rtk)?
2. Extending the surface: if a user wanted rtk to reach more of this workload, what's the path? The TOML filter pipeline (.rtk/filters.toml) is user-extensible, but from the source it only fires when the agent already invokes rtk <cmd> — the static rewrite hook doesn't consult it. Is there a supported way to register new base commands (e.g. forge's node .forge/tools/*.cjs) for automatic rewriting without recompiling?
3. Tool output vs commands: is there any supported path for rtk to compress tool output that didn't come from a Bash command it can re-invoke (file reads, another server's MCP results), or is that out of scope by design?
4. Anything in our setup you'd consider an unfaithful integration?

Full protocol, raw transcripts, and per-run data are public, and we'll quote your response verbatim in the writeup. The issue link with the full setup is above; happy to look at any single run where the numbers don't line up.

[aeppling]

Hello @devasur

You're totally right about the RTK impact surface which for now, only target bash commands output.
The reason is simple here, we want to cover this properly before extending to new compressions methods and surfaces, because a bad compression will create MORE token than it save, and this has been observed with our Benchmark on RTK but also on other solution which claim saving with MCP and a lot of various methods.

Where you're benchmark is for me "weird", is for example as you say the input token variance et the per cost inputs.

RTK Does not inject mcp or others and have a really slim context, so RTK adding token per turns is not possible (also because of cache, so any initial instructions is cached and not billed every call).

Another important point, we have our internal benchmark, not fully complete, that we will publish soon.

The reason why it is not published yet is also part of why your benchmark could not be always representative.

I had an hard time evaluating RTK because of the variance in the agent behavior, even when running 20 sessions with same prompt without RTK, you'll observe a big variance in all token source and outputs.
On my side, to benchmark our solution, we for now give an instructions path on what to execute, but telling the agent to expect the information he want, if he does not get it, he can retry with another commands.

This allow to identify where RTK can over-compress and cost more api turn (which is basically the challenge, compressing without causing new api turn to get the truncated information for exemple) while not falling in all the variance.

For your setup, just ensure using an API that use cache and ensure RTK is using the pure hook to rewrite, if using local models with no cache, this could of course change a lot, especially with other solution you benchmarked, since they'll re-inject the context / mcp.

Summary

We're open to benchmark that could help grow RTK, but this has to be clean and if it is a comparaison, should be compared on the correct task.

RTK perform really well on all dev commands that aim testing, building, etc.

But on a edit / write task, RTK has no interest of being benchmarked, since it does not operate fully on this surface yet.
Else we need a full task list with each solution and how they perform for each, else we prefer not to be compared on a task that benefits most to others solutions.

Last thoughts, you're talking about "independant" benchmark, but also working on forge and forge-compress , that could be a direct competitor to the solution you benchmark (leanCtX, RTK, ....)

Blockers: No defined variance that could tell above what value it is meaningful, no way to understand why your benchmark say RTK pay more token per turn, which look not possible (RTK if not correct will make you pay more because additional turns rather than pay more per turn), only one task that does not fit all your benchmark candidat and cover only a surface,

more important, remove "independant", since you aim to also build your solution , that just make this shady.

Thanks for asking us first ! If you find any interesting results for RTK we can use to enhance, we are of course happy to take it !

[aeppling]

If needed we can discuss more here or on the RTK discord, of course open to discuss with your so you can provide a benchmark we will agree on.

[devasur]

Thanks @aeppling — this is genuinely useful, and a few of your points let me tighten the writeup. Taking them in order:

On per-turn cost — I think we agree. I want to be clear we are not claiming rtk adds tokens per turn. The comment above says rtk is "mechanically incapable of increasing the token bill," that the +30% is "run-path variance, not rtk overhead," and that rtk's lowest run lands at native (−1.6%). So your "rtk adding token per turn is not possible" is exactly our read — the headline number is explicitly not attributed to rtk. If anything in our phrasing suggested otherwise, point me at it and I'll fix it.

On cache — correct, and deliberate. This rail is a non-caching, request-metered provider (ollama-cloud) on purpose: it isolates each product's mechanism by removing cache effects that would otherwise dominate. On a cache-priced API your cached instructions aren't re-billed, which only helps rtk — so this rail is, if anything, the harder setting for the products and we scope every finding to "this harness / this provider." That scoping is in the writeup; I'll make it more prominent.

On task surface — that's the finding, not a knock. You're right that rtk isn't built for edit/write surface yet, and that's precisely what we report: on this SDLC workload, registry-covered bash output is ~2.5% of the bill, so rtk's (real, 64–76%) savings have a small surface here. We frame this as architecture–surface fit, and we do not rank products against each other on a shared task — each is measured only against native on the same harness. rtk "performed as designed" is a direct quote from our notes.

On variance — this is exactly why the protocol is multi-run. You describe big run-to-run variance even across 20 native-only sessions with the same prompt — we see the same thing, and it's precisely why the protocol runs every arm multiple times (n=3 per arm) and reports the median, not a single session. The pre-registered, frozen protocol and every per-run transcript are public so the spread is visible, not hidden behind one number. I agree variance is large relative to the effects here and the writeup says so; when your internal benchmark publishes I'll link it, and I'll include any response you write here verbatim.

On "independent." The conflict is disclosed up front (issue + README), so I'd push back on "shady" — but let me address the substance, because it's a fair thing to probe:

• forge-cli is the harness, not a benchmarked product. It's the runtime every arm executes on, native included — identical for all three vendors. It isn't an arm in the comparison.
• forge-compress only compresses forge's own store artifacts (the harness's tracking documents), and it does so uniformly across every arm, native included. It is not in rtk's surface and gives no arm a relative advantage — it's held constant. The per-run data showing this is public.
• "Independent" here means no vendor funded, commissioned, or reviewed-for-approval this work — not "the author ships nothing in this space." I think that's the honest use of the word alongside a disclosed conflict, but I'm open to wording that makes the distinction unmistakable.

The raw transcripts and per-run numbers behind all of the above are in the repo — happy to walk any single run where something doesn't line up, here or on Discord. And the offer stands: your response goes into the publication verbatim.

[devasur]

@aeppling — the writeup is live, and thank you for engaging with it as directly as you did. The scrutiny was fair and the piece is better for it: your points on variance and task surface are reflected as our own findings, not treated as objections to wave away.

We've soft-published — direct link only, not yet on our homepage or series index — to give you a review-and-comment window before broader release:
https://entelligentsia.in/blog/who-owns-the-context-window-the-three-meters

What's relevant to you: the post states that rtk worked exactly as designed (64–76% saved on the commands it touches) and that the +30% median is run-path variance, not an rtk cost — rtk has no per-turn token-adding mechanism, and its floor run sits at native. The ~1–2.5% addressable surface is framed as architecture–surface fit, not product quality. On "independent": we keep the term and explain why — Forge here is the neutral harness applied identically to every arm including the native baseline, forge-compress touches only Forge's own store artifacts held constant across arms, and the conflict is disclosed. Your reply is represented in context.

Review-and-comment window through June 22. If you feel anything is unfair or off, tell us and we'll correct it before broad release — and if you'd like a clean standalone statement quoted verbatim, send it over and we'll include it as-is. Everything is public and reproducible. Thanks again for the time and the candor.

[devasur]

@aeppling — heads-up that we're planning a second round (R2), and rtk is invited if you're interested. Two of your points from the first round are baked into the R2 design, so this should be a fairer surface for rtk than the first task was:

• Cache-priced rail. You argued the non-caching ollama-cloud rail wasn't representative. R2 runs on a cache-priced Anthropic rail (opus/sonnet/haiku via pi), where the economics are different — that's the deployment you pointed us at.
• Dev-command surface, not edit/write. You said rtk should be measured on dev commands, not file edits. R2's task is a /forge:fix-bug on a real git C source — a planted one-token versioncmp bug, oracle removed (variant of https://entelligentsia.in/blog/capability-test-coding-agent). It is build/test/shell-heavy (make, the git test suite, grep/localization) — i.e. exactly the dev-command payload rtk rewrites, on a much larger codebase than the first task.

Other R2 facts: latest software from every product, phase-isolated, k=3 per arm, capability-gated (a run is only priced if it actually lands the correct fix, judged by rebuild + reproduce + suite — so "worked as designed" can't be undercut by a failed run). Headline is per-arm cost on the cache rail; loops/turns are recorded.

Two asks so the rtk arm is faithful and reproducible:

1. Pin the rtk version you want tested (a specific public release/tag — please name it, not "latest", since we freeze each arm's version for the run).
2. Share the config/setup steps for the rtk arm on this testbench — how you want rtk wired to rewrite pi's tool calls (the integration we used in R1, or whatever you now recommend), plus any flags. If it's easier to express as a PR or a short script against latest tokbench, great — we'll run it as-is so "installed = running as designed."

If you have a preflight check that confirms rtk is actually rewriting commands in this setup (so we don't repeat any "engaged but not on the addressable surface" ambiguity), we'll run it before any priced run and treat green = ready.

Same standing offer as before: verbatim right-of-reply in the writeup, and if your internal benchmark is publishable we'll link it. You can check the latest tokbench for the R2 spec as it firms up. Thanks.

[Yohjisakamoto]

This independent agentic-workload benchmark reinforces why end-to-end agent outcomes matter more than compressed text alone: DeepSWE found Tura cuts coding-agent rounds by 89% and total token cost by 80%; would you evaluate the methodology? https://turaai.net/docs#benchmark-current-test-set-record