feat(chat): throttle chat UI updates by default - #2058
Open
ben-reitz wants to merge 1 commit into
Open
Conversation
🦋 Changeset detectedLatest commit: f8be3ad The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
ben-reitz
force-pushed
the
fix/default-chat-throttle
branch
from
August 6, 2026 14:06
1fdc907 to
60edd64
Compare
ben-reitz
marked this pull request as ready for review
August 6, 2026 14:10
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
agents
@cloudflare/ai-chat
@cloudflare/codemode
create-think
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
This was referenced Aug 6, 2026
gr2m
added a commit
to vercel/ai
that referenced
this pull request
Aug 6, 2026
## Background `useChat` currently throttles its messages subscription callback, but its `useSyncExternalStore` snapshot always reads the latest `chat.messages` array. Because streaming replaces that array for every chunk, any unrelated React render can observe a new snapshot before the throttled callback publishes it. In high-frequency streams this bypasses `throttle`, causes per-chunk renders, and can contribute to the "Maximum update depth exceeded" failures reported in #6166. ## Summary - Keep a published messages snapshot per `useChat` hook and advance it from that hook's throttled subscription callback. - Publish the latest message snapshot before `ready` or `error` becomes observable, including normal completion and aborts. - Synchronize updates that occur between render and subscription, and ignore delayed callbacks after a hook unsubscribes or changes chat instances. - Add regression coverage for unrelated renders, terminal status/message coherence, aborts, errors, and delayed callbacks after chat replacement. - Turn the existing Next.js throttle route into a deterministic end-to-end reproduction that streams 500 chunks while forcing unrelated renders, reports pass/fail from observed snapshot identities, and verifies the complete message is visible when status becomes `ready`. - Add a patch changeset for `@ai-sdk/react`. ## Contributor Credit - @brahmveda-arkin reported #6166. - @takumiz19 isolated the snapshot/subscription mismatch and proposed #17893. - @ben-reitz demonstrated the practical value of a conservative UI update cadence in cloudflare/agents#2058. ## End-to-End Verification Ran `/chat/throttle` in `examples/ai-e2e-next` in a real browser. The route streams 500 chunks (1,000 assistant characters) with `throttle: 50` while a zero-delay timer independently re-renders the component. - Before: **FAIL**, 255 distinct message snapshots in 1,475ms (maximum expected: 34), across 946 total React renders. - After: **PASS**, 15 distinct message snapshots in 1,144ms (maximum expected: 27), across 602 total React renders. The patched run rendered all 1,000 assistant characters on the same render where status became `ready`, and had no Next.js error overlay or browser error. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work This PR intentionally leaves the current opt-in default unchanged, so it does not protect applications that omit `throttle` from high-frequency unthrottled rendering. For v8, I recommend making a 50ms UI publication cadence the default when `throttle` is omitted, with `throttle: 0` as the explicit unthrottled opt-out. Stream processing, tool handling, and callbacks should remain immediate; only snapshots exposed to React should be paced. The default should also guarantee an immediate leading publication and a terminal flush so the final messages and `ready` status stay coherent. This would cap the normal rendering rate at about 20 updates per second and protect applications that do not know they need to opt in today. The tradeoff is up to 50ms of additional visible text latency and an explicit opt-out for applications that intentionally need per-chunk rendering, which makes the behavior change appropriate for a major release. ## Related Issues Addresses the throttled snapshot bypass discussed in #6166. Reports using the default unthrottled behavior remain outside this PR. Closes #17893. Related to cloudflare/agents#2058.
gr2m
added a commit
to vercel/ai
that referenced
this pull request
Aug 6, 2026
## Background `useChat` in AI SDK 6 throttles its messages subscription callback, but its `useSyncExternalStore` snapshot always reads the latest `chat.messages` array. Because streaming replaces that array for every chunk, an unrelated React render can observe a new snapshot before the throttled callback publishes it. In high-frequency streams this bypasses `experimental_throttle`, causes per-chunk renders, and can contribute to the "Maximum update depth exceeded" failures reported in #6166. This is a branch-native backport of #18525 to `release-v6.0`. ## Summary - Keep a published messages snapshot per `useChat` hook and advance it from that hook's throttled subscription callback. - Publish the latest message snapshot before `ready` or `error` becomes observable, including normal completion and aborts. - Synchronize updates that occur between render and subscription, and ignore delayed callbacks after a hook unsubscribes or changes chat instances. - Add regression coverage for unrelated renders, terminal status/message coherence, aborts, errors, and delayed callbacks after chat replacement. - Turn the existing Next.js throttle route into a deterministic end-to-end reproduction that streams 500 chunks while forcing unrelated renders, reports pass/fail from observed snapshot identities, and verifies the complete message is visible when status becomes `ready`. - Update the reproduction's stale text stream-part shape to the v6 protocol and avoid a render-counter hydration mismatch. - Add a patch changeset for `@ai-sdk/react`. ## Contributor Credit - @brahmveda-arkin reported #6166. - @takumiz19 isolated the snapshot/subscription mismatch and proposed #17893. - @ben-reitz demonstrated the practical value of a conservative UI update cadence in cloudflare/agents#2058. ## Manual Verification Ran `/chat/throttle` in `examples/ai-e2e-next` in a real browser. The route streamed 500 chunks (1,000 assistant characters) with `experimental_throttle: 50` while a zero-delay timer independently re-rendered the component. The patched v6 run passed with 15 distinct message snapshots in 860ms (maximum expected: 22), across 462 total React renders. All 1,000 assistant characters were visible on the first render where status became `ready`, with no Next.js error overlay. Also ran: - `pnpm --filter @ai-sdk/react test -- use-chat.ui.test.tsx` (60 tests passed) - `pnpm check` - `pnpm type-check:full` ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work This backport intentionally leaves the v6 opt-in default unchanged, so applications must continue to set `experimental_throttle` to benefit from paced React publications. For v8, I recommend making a 50ms UI publication cadence the default when `throttle` is omitted, with `throttle: 0` as the explicit unthrottled opt-out. Stream processing, tool handling, and callbacks should remain immediate; only snapshots exposed to React should be paced. The default should guarantee an immediate leading publication and a terminal flush so final messages and `ready` status stay coherent. This would cap the normal rendering rate at about 20 updates per second and protect applications that do not know they need to opt in today. The tradeoff is up to 50ms of additional visible text latency and an explicit opt-out for applications that intentionally need per-chunk rendering, which makes the behavior change appropriate for a major release. ## Related Issues Backport of #18525. Addresses the throttled snapshot bypass discussed in #6166. Reports using the default unthrottled behavior remain outside this PR. Related to cloudflare/agents#2058.
gr2m
added a commit
to vercel/ai
that referenced
this pull request
Aug 6, 2026
## Background `useChat` in AI SDK 5 throttles its messages subscription callback, but its `useSyncExternalStore` snapshot always reads the latest `chat.messages` array. Because streaming replaces that array for every chunk, an unrelated React render can observe a new snapshot before the throttled callback publishes it. In high-frequency streams this bypasses `experimental_throttle`, causes per-chunk renders, and can contribute to the "Maximum update depth exceeded" failures reported in #6166. This is a branch-native backport of #18525 to `release-v5.0`. ## Summary - Keep a published messages snapshot per `useChat` hook and advance it from that hook's throttled subscription callback. - Publish the latest message snapshot before `ready` or `error` becomes observable, including normal completion and aborts. - Synchronize updates that occur between render and subscription, and ignore delayed callbacks after a hook unsubscribes or changes chat instances. - Add regression coverage for unrelated renders, terminal status/message coherence, aborts, errors, and delayed callbacks after chat replacement. - Turn the existing Next.js throttle route into a deterministic end-to-end reproduction that streams 500 chunks while forcing unrelated renders, reports pass/fail from observed snapshot identities, and verifies the complete message is visible when status becomes `ready`. - Update the reproduction's stale text stream-part shape to the v5 protocol and avoid a render-counter hydration mismatch. - Add a patch changeset for `@ai-sdk/react`. ## Contributor Credit - @brahmveda-arkin reported #6166. - @takumiz19 isolated the snapshot/subscription mismatch and proposed #17893. - @ben-reitz demonstrated the practical value of a conservative UI update cadence in cloudflare/agents#2058. ## Manual Verification Ran `/use-chat-throttle` in `examples/next-openai` in a real browser. The route streamed 500 chunks (1,000 assistant characters) with `experimental_throttle: 50` while a zero-delay timer independently re-rendered the component. The patched v5 run passed with 15 distinct message snapshots in 1,227ms (maximum expected: 29), across 654 total React renders. All 1,000 assistant characters were visible on the first render where status became `ready`, with no Next.js error overlay. Also ran: - `NODE_PATH=packages/rsc/node_modules pnpm --filter @ai-sdk/react test -- use-chat.ui.test.tsx` (53 tests passed; v5's React Vitest config imports the plugin already pinned by the RSC package but does not declare it itself) - `pnpm --filter @ai-sdk/react type-check` - `pnpm --filter @ai-sdk/react... build` - `pnpm check` - `git diff --check` `pnpm type-check:full` remains red in this checkout because unchanged v5 examples resolve incompatible React type versions. It did not report any changed file; the changed React package type-check and declaration build pass, and the changed Next.js route compiled successfully during browser verification. ## Tasks - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work This backport intentionally leaves the v5 opt-in default unchanged, so applications must continue to set `experimental_throttle` to benefit from paced React publications. For v8, I recommend making a 50ms UI publication cadence the default when `throttle` is omitted, with `throttle: 0` as the explicit unthrottled opt-out. Stream processing, tool handling, and callbacks should remain immediate; only snapshots exposed to React should be paced. The default should guarantee an immediate leading publication and a terminal flush so final messages and `ready` status stay coherent. This would cap the normal rendering rate at about 20 updates per second and protect applications that do not know they need to opt in today. The tradeoff is up to 50ms of additional visible text latency and an explicit opt-out for applications that intentionally need per-chunk rendering, which makes the behavior change appropriate for a major release. ## Related Issues Backport of #18525. Addresses the throttled snapshot bypass discussed in #6166. Reports using the default unthrottled behavior remain outside this PR. Related to cloudflare/agents#2058.
9 tasks
Streaming writes chat state once per chunk, and each write re-renders. A burst of chunks, such as a resumed stream replaying a long turn, reaches React's 50-render limit and throws. The AI SDK reports that as a failed turn even though the server completed it (#1913). Merging replayed chunks removes one update per chunk but still costs one update per part, so a turn with enough tool steps fails anyway. A throttle does not depend on how many chunks arrive: any value above zero prevents the crash, because the update then arrives from a timer. The default is 50ms. Measured over a 200-chunk turn at ~100 chunks/sec, that removes 78% of renders (404 commits to 90), and larger values save progressively less while lagging further behind the stream. It is also the value the AI SDK documents. The first chunk is never delayed, because the SDK throttles with throttleit, which runs the first call of an idle window immediately. Both option names carry the value: @ai-sdk/react v3 only reads experimental_throttle, v4 reads throttle, and both majors are in our peer range. Passing `throttle: false` omits both names instead of sending 0, since both majors decide with `waitMs != null` and only an omitted option reaches their unthrottled path.
ben-reitz
force-pushed
the
fix/default-chat-throttle
branch
from
August 7, 2026 07:24
60edd64 to
f8be3ad
Compare
Contributor
Author
|
☝🏼 Looks like they're actually turning on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A proposal, stacked on #2050. Fixes #1913 together with that PR.
Why a second change
#2050 stops a resumed stream from writing chat state once per replayed chunk. It merges the chunks that belong to the same piece of text, so a few hundred chunks become a handful of writes.
That helps enormously, but it only merges within a part. A tool step costs about seven writes on its own, so a replayed turn of more than about seven tool steps still reaches React's 50-render limit and still shows a false error. Measured with the merge in place: six steps fine, eight throws.
A throttle does not care how many chunks arrive, so it covers what merging cannot.
The change
useAgentChatnow coalesces chat updates every 50ms. Passthrottle: falseto render every chunk as it arrives, or a number to pick your own interval.Nothing else changes, and the first chunk of a stream is never delayed - the AI SDK throttles with
throttleit, which runs the first call of an idle window immediately.Why 50ms
Any value above zero prevents the crash, because the update then arrives from a timer rather than from the current task. Verified at 1, 4, 16, 50, 100 and 250ms. So the size of the number is a cost decision rather than a safety one. Measured over a 200-chunk turn at ~100 chunks/sec (caveat that this is obvs on a powerful macbook):
50ms removes 78% of the renders. Larger values save progressively less while letting the text lag further behind the stream. It is also the value the AI SDK's own documentation uses, and 25 of our 27 examples already pass 100ms by hand, so throttling is not new behaviour for our own UIs - only the default is.
Two things worth knowing
@ai-sdk/reactv3 reads onlyexperimental_throttle; v4 renamed it tothrottle. Both majors are in our peer range, so the value is sent under both names. Turning it off omits both names rather than sending0, because both majors decide withwaitMs != null, and only an omitted option reaches their unthrottled path.This proposed change provides a mitigation rather than a guarantee.
useChatthrottles its store subscription, but itsgetSnapshotreturns a new messages array on every chunk, and React forces a synchronous re-render whenever that identity moved during a render - a path the throttle never sees. That is vercel/ai#6166, with a fix open in vercel/ai#17893. Writing state less often is the only thing that bounds it, which is why #2050 does the heavier lifting.Tests
Eight unit tests in
chat/__tests__/chat-throttle.test.tscover the precedence rules and both spellings. Two tests inreact-tests/default-throttle.test.tsxdrive the real hook with a 12-step tool turn: it stays healthy on defaults, andthrottle: falserestores the old failure, which proves the caller's value reaches the SDK.Suites: agents chat 540 · react 135 · ai-chat 737 · think 920 + react 5 ·
pnpm run check.