Skip to content

Commit

Permalink
Optimize callbacks TransformStream (#34)
Browse files Browse the repository at this point in the history
Co-authored-by: Jared Palmer <jared@jaredpalmer.com>
  • Loading branch information
jridgewell and jaredpalmer committed Jun 8, 2023
1 parent fc83e95 commit 2c6fa04
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 11 deletions.
5 changes: 5 additions & 0 deletions .changeset/loud-maps-invent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"ai-connector": patch
---

Optimize callbacks TransformStream to be more memory efficient when `onCompletion` is not specified
18 changes: 7 additions & 11 deletions packages/core/src/ai-stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,26 +53,22 @@ export function createCallbacksTransformer(
const encoder = new TextEncoder()
let fullResponse = ''

const { onStart, onToken, onCompletion } = callbacks || {}

return new TransformStream<string, Uint8Array>({
async start(): Promise<void> {
if (callbacks?.onStart) {
await callbacks.onStart()
}
if (onStart) await onStart()
},

async transform(message, controller): Promise<void> {
controller.enqueue(encoder.encode(message))

if (callbacks?.onToken) {
await callbacks.onToken(message)
}
// TODO: If `onCompletion` isn't defined, then we could skip this and save memory.
// This is very likely to receive rope-concat optimizations, so at least it's not slow
fullResponse += message
if (onToken) await onToken(message)
if (onCompletion) fullResponse += message
},

flush() {
return callbacks?.onCompletion?.(fullResponse)
async flush(): Promise<void> {
await onCompletion?.(fullResponse)
}
})
}
Expand Down

1 comment on commit 2c6fa04

@vercel
Copy link

@vercel vercel bot commented on 2c6fa04 Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

ai-sdk-doc – ./

sdk.vercel.ai
ai-utils-docs.vercel.sh
ai-sdk-doc.vercel.sh
ai-sdk-doc-git-main.vercel.sh

Please sign in to comment.