ai-bot: add AI_BOT_STREAMING_MODE=off to skip mid-stream Matrix edits#5556
Conversation
In `off` mode the responder emits only the initial thinking placeholder and the final consolidated `m.replace` edit at stream end, cutting room events per response from ~600 to 2. Default (`room-edits`) is unchanged. Cancellation and final-event semantics (isStreamingFinished, isCanceled, continuation chaining) route through the same path in both modes — only the throttled onChunk send is gated. Phase 1 of CS-12124. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces an AI_BOT_STREAMING_MODE environment variable in the ai-bot responder to allow disabling mid-stream Matrix room edits, aiming to reduce the volume of m.replace events during a response.
Changes:
- Add
AI_BOT_STREAMING_MODEparsing inResponder(defaulting to current “room-edits” behavior). - Gate per-chunk throttled sends in
offmode so intermediatem.replaceedits are skipped. - Add a new
Respondingtest assertingoffmode produces only the thinking placeholder plus a single final consolidated event.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| packages/ai-bot/lib/responder.ts | Adds streamingMode env var handling and gates per-chunk message sends in off mode. |
| packages/ai-bot/tests/responding-test.ts | Adds a regression test for off mode ensuring no intermediate sends occur and the final event is consolidated. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (responseStateChanged && this.streamingMode !== 'off') { | ||
| await this.sendMessageEventWithThrottling(); | ||
| } |
There was a problem hiding this comment.
[Claude Code 🤖] Confirmed and fixed in 5cf06f8. The mid-turn gate at this line was correct; the bug was upstream — onChunk was flipping responseState.isStreamingFinished from the terminal chunks finish_reason: "stop", so finalize saw no transition and its send path was skipped too. Fix defers that flag transition to finalize()whenshouldStreamMidTurn` is false, so the final consolidated event always lands via the flush path.
| for (let i = 0; i < 10; i++) { | ||
| await responder.onChunk({} as any, snapshotWithContent('content ' + i)); | ||
| } |
There was a problem hiding this comment.
[Claude Code 🤖] Good catch — updated in 5cf06f8. The 10th onChunk now carries choices: [{ finish_reason: "stop", ... }], which reproduces the missing-final-event bug without the fix and passes with it.
jurgenwerk
left a comment
jurgenwerk
left a comment
There was a problem hiding this comment.
Looks ok but the copilot review seems important to address
Summary
AI_BOT_STREAMING_MODEenv var topackages/ai-bot/lib/responder.ts. Default isroom-edits(current behavior);offskips the throttled per-chunkm.replaceedit and only emits the initial thinking placeholder plus the final consolidated event at stream end.offmode.isStreamingFinished,isCanceled, continuation chaining for >16kB bodies) route through the same finalize/flush path in both modes — only theonChunkcall is gated.Context
Phase 1 of CS-12124, the plan for the Improve Synapse Performance by Killing Streaming project. Synapse was observed at 125% CPU with a single active AI Assistant user, driven by ~600
m.replaceedits per response. This PR ships the lowest-risk mode — pure "no streaming" — so we can measure the immediate CPU relief and dogfood the fallback UX while CS-12263 adds streaming back over Matrix'ssendToDevicetransport.The
to-devicevalue is accepted by the getter but behaves likeroom-editstoday; the substitution comes in CS-12263.Test plan
Respondingtest: inoffmode, 10 rapid chunks +finalize()produce exactly 2 events (thinking + final) withisStreamingFinished=trueandm.replacepointing at the placeholder.Respondingtests pass.Responder Cancellationtests pass —finalize({isCanceled:true})still emits the finalisCanceledevent through the same path.AI_BOT_STREAMING_MODE=offon the ai-bot task, run a chat turn, verify only 2 room events land per turn and Synapse CPU drops on the dashboard.