Skip to content

test(server): budget ACP tool call emissions at the adapter boundary - #122

Merged
rynfar merged 1 commit into
pylonfrom
test/acp-emission-budget
Aug 27, 2026
Merged

test(server): budget ACP tool call emissions at the adapter boundary#122
rynfar merged 1 commit into
pylonfrom
test/acp-emission-budget

Conversation

@rynfar

@rynfar rynfar commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Closes a coverage gap I hit while reviewing #118, where I could describe a websocket-traffic risk but not measure it.

The gap

apps/server/src/server.test.ts already enforces a thread transfer budget — five metrics per provider, hard ceilings, CI fails on a violation. But it covers codex and claudeAgent only, and it drives them by queueing recorded fixtures into an adapter harness rather than starting a real adapter.

So two things are invisible to it. ACP providers aren't in the provider list, and no adapter runs at all — which means decideToolCallUpdateEmission, the function that decides how many provider events a tool call generates in the first place, is never executed. That function is shared by Grok, Cursor, and Prime Agent.

Adding an ACP provider to TRANSFER_BUDGETS would not have fixed this. It would measure the same fixture-driven pipeline under a different label and still never reach the emission decision. The measurement has to happen at the adapter boundary.

What this adds

A T3_ACP_STREAM_COMMAND_CHUNKS scenario in the ACP mock agent that streams a long-running command the way a real terminal tool does: one tool_call_update per chunk, cumulative text on rawOutput.stdout, and a title that never changes. Then a budget over it, driven through the Grok adapter and asserting on emitted ProviderRuntimeEvents.

The budget is two-sided, and that is the point. A ceiling alone would have passed on the exact bug #118 fixed: because command tools keep detail equal to the command, coalescing that measured only detail saw no growth and suppressed every in-progress update, delivering the whole build log at completion. To a ceiling that reads as "wonderfully few messages." To a user it is a dead UI during a build.

Verification

For 40 chunks of 64 characters — 2,560 characters of stdout:

Behavior In-progress updates Budget result
Current (pylon) 11 passes — 10 growth flushes plus the status change, matching the 256-char cadence
Pre-#118 emission logic 1 fails the floor — everything withheld until completion

I confirmed the second row by temporarily restoring the old detail-only coalescing in AcpRuntimeModel.ts and running the budget against it, then restoring the file. So this test is demonstrably capable of catching the regression it exists for, rather than merely passing today.

Bounds are 4–16 against an observed 11, deliberately loose. Tuning the coalescing thresholds should not fail this test; losing streaming or losing coalescing should.

  • 104 tests pass across the new budget plus the Grok, Cursor, shared-ACP runtime, session runtime, JSON-RPC, and CLI-probe suites, confirming the new mock scenario is inert for existing tests (it is gated on a flag defaulting to 0).
  • t3 typecheck clean, lint clean on both files.

No production code changes — mock agent and test only.

Model: Claude Opus 5. Harness: Claude Code.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The thread transfer budget in server.test.ts caps HTTP and WebSocket bytes, but
it replays recorded fixtures through Codex and Claude. Neither is an ACP
provider, and the harness never starts an adapter, so nothing measures how many
provider events an ACP tool call produces in the first place. That decision is
`decideToolCallUpdateEmission`, shared by Grok, Cursor, and Prime Agent, and it
was until recently withholding every in-progress update for command tools.

Adds a streaming-command scenario to the ACP mock agent and a two-sided budget
over it. The ceiling keeps coalescing honest. The floor is the part that
matters: a ceiling alone reads "withheld everything" as a pass.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Aug 27, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.3 KiB 13.3 KiB +30 B (+0.2%) 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 6.9 KiB −4 B (−0.1%) 7.3 KiB
Codex Live turn WebSocket wire 6.4 KiB 6.4 KiB +34 B (+0.5%) 7.8 KiB
Codex Live turn WebSocket decoded 55.5 KiB 55.6 KiB +44 B (+0.1%) 66.4 KiB
Codex Live turn messages 9 10 +1 (+11.1%) 21
Claude Total thread wire 13.3 KiB 13.3 KiB −24 B (−0.2%) 15.1 KiB
Claude Thread snapshot wire 6.9 KiB 6.9 KiB +5 B (+0.1%) 7.3 KiB
Claude Live turn WebSocket wire 6.4 KiB 6.4 KiB −29 B (−0.4%) 7.8 KiB
Claude Live turn WebSocket decoded 56.4 KiB 56.3 KiB −44 B (−0.1%) 66.4 KiB
Claude Live turn messages 10 9 −1 (−10.0%) 21

Baseline: 5906db9 · PR result: c79fbe5 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 109.4 KiB
  • Claude decoded thread snapshot: 110.1 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@rynfar
rynfar merged commit 08e9acc into pylon Aug 27, 2026
14 checks passed
@rynfar
rynfar deleted the test/acp-emission-budget branch August 27, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant