-
Notifications
You must be signed in to change notification settings - Fork 2.5k
fix(openai-compatible): return actual usage when steaming instead of NaN #6648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ki7ick
wants to merge
197
commits into
vercel:main
Choose a base branch
from
ki7ick:fix/usage-return-nan-fix
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…nge (vercel#5531) Co-authored-by: Carl Brugger <cebrugg@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…#5541) Co-authored-by: Bram Meerten <bram.meerten@acagroup.be>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
# Releases ## ai@4.3.15 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/react@1.2.12 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/amazon-bedrock@2.2.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/anthropic@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/azure@1.3.23 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai@1.3.22 ## @ai-sdk/cerebras@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/cohere@1.2.10 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/deepinfra@0.2.15 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/deepseek@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/elevenlabs@0.0.3 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/fal@0.1.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/fireworks@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/google@1.2.17 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/google-vertex@2.2.21 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/anthropic@1.2.11 - @ai-sdk/google@1.2.17 ## @ai-sdk/groq@1.2.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/hume@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/lmnt@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/luma@0.1.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/mistral@1.2.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/openai@1.3.22 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/openai-compatible@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/perplexity@1.1.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/provider-utils@2.2.8 ### Patch Changes - d87b9d1: fix(provider-utils): fix SSE parser bug (CRLF) ## @ai-sdk/react@1.2.12 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/replicate@0.2.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/revai@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/solid@1.2.13 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/svelte@2.1.12 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/togetherai@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/ui-utils@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/valibot@0.1.27 ### Patch Changes - ai@4.3.15 ## @ai-sdk/vue@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/xai@1.2.16 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ming stream (vercel#6224) ## Background This pull request updates the documentation for the GET handler in the resumable streams section to account for a race condition that may occur when the stream ends in-flight. ## Summary Updated the route by adding the code snippet that handles the above condition. ## Tasks - [x] Documentation has been added / updated (for bug fixes / features) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Related Issues vercel#6052
## Background Bug with Vue that led to status not updating when tab was changed. ## Summary Changes status from using SWR to using Vue ref.
# Releases ## @ai-sdk/vue@1.2.12 ### Patch Changes - b54cb59: fix (ai-sdk/vue): fix status reactivity Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background In mcp doc, `mcpClient.tools()` need to await ## Verification Just fix doc problem Co-authored-by: georgezouq <george@staringos.com> Co-authored-by: Lars Grammel <lars.grammel@gmail.com>
## Background [Vertex now supports extraction of thinking tokens in certain Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking). When the configuration is passed via `providerOptions`, the sdk: 1. Did not extract reasoning tokens 2. Did not pass `include_thoughts` to the provider ## Summary Added extraction logic to google-generative-ai package to parse reasoning tokens. Added a `includeThoughts` switch to the `thinkingConfig` for vertex models. ## Verification I verified it manually. Testable via examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily copiable to google provider. ## Related Issues Fixes vercel#6259
## Background Release requires change to `google` provider package. ## Summary Add changeset.
# Releases ## @ai-sdk/google@1.2.18 ### Patch Changes - 4b2e1b0: Add reasoning token output support for gemini models via Vertex AI Provider ## @ai-sdk/google-vertex@2.2.22 ### Patch Changes - fe24216: Add reasoning token output support for gemini models via Vertex AI Provider - Updated dependencies [4b2e1b0] - @ai-sdk/google@1.2.18 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
) ## Background Like many other OpenTelemetry instrumentations, Laminar depends on `@opentelemetry/instrumentation`, which, in turn, depends on `require-in-the-middle` and `import-in-the-middle`. Importing and initializing Laminar inside Next.js `instrumentation.ts` file causes Next.js to try resolving these two packages, but fails, and results in: - Laminar not being able to send traces. This is because unlike many other instrumentation libraries, Laminar is not intrusive and does not set its tracer provider globally (so that others, e.g. `@vercel/otel` can set theirs). - Error messages (see below) We have tried many different things to debug, including bundling Laminar differently, shipping those two packages within Laminar as `noExternal`, adding a separate entrypoint in our package for Next.js, but nothing seems to have worked. The only thing that's worked was adding `@lmnr-ai/lmnr` in `serverExternalPackages` in [next.config](https://nextjs.org/docs/app/api-reference/config/next-config-js/serverExternalPackages). ## Summary Add a subsection within the Next.js section that describes
…#6315) ## Background It is not clear what AI SDK version user-submitted issues refer to. ## Summary Add version section to issue template.
## Background Integrating with Patronus! ## Summary A new Patronus.mdx file showing how logs can be imported into Patronus via OTel. ## Tasks - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) --------- Co-authored-by: Snigdha Banda <snigdhabanda@Snigdhas-MacBook-Pro.local> Co-authored-by: nicoalbanese <gcalbanese96@gmail.com> Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com>
## Summary Announce v5 Alpha ## Tasks - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root)
…spans (vercel#6357) ## Background `generateObject`, `generateText`, `streamText`, and `streamObject` currently call `JSON.stringify` on the input messages. If the input messages contain an image, it is most likely normalized into a `Uint8Array`. `JSON.stringify` does not the most obvious things to TypedArrays including `Uint8Array`. ```javascript // this returns '{"0": 1,"1": 2,"2": 3}', where I'd expect this to be '[1,2,3]' JSON.stringify(new Uint8array([1, 2, 3])) ``` In practice, this results in bloating images by about 5-15x depending on the original image size. For Laminar, for example, a span with 3 avg sized images will not be able to be sent as it is larger than the (reasonably high) gRPC payload size for our traces endpoint. From [MDN docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#examples): ```javascript // TypedArray JSON.stringify([new Int8Array([1]), new Int16Array([1]), new Int32Array([1])]); // '[{"0":1},{"0":1},{"0":1}]' JSON.stringify([ new Uint8Array([1]), new Uint8ClampedArray([1]), new Uint16Array([1]), new Uint32Array([1]), ]); // '[{"0":1},{"0":1},{"0":1},{"0":1}]' JSON.stringify([new Float32Array([1]), new Float64Array([1])]); // '[{"0":1},{"0":1}]' ``` ## Summary Added a function that maps over messages in a `LanguageModelV1Prompt` and maps over content parts in each message, replacing `UInt8Array`s with raw base64 strings instead. Call this function when calling `recordSpan` for the inner (doStream/doGenerate) span in `generateObject`, `generateText`, `streamText`, and `streamObject`. ## Verification Ran this small script against a local instance of Laminar and logged the Telemetry payloads (span attributes) on the backend to verify that they are indeed base64. ```javascript import { Laminar, getTracer } from '@lmnr-ai/lmnr' Laminar.initialize(); import { openai } from '@ai-sdk/openai' import { generateText, generateObject, streamText, streamObject, tool } from "ai"; import { z } from "zod"; import dotenv from "dotenv"; dotenv.config(); const handle = async () => { const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/b/bc/CoinEx.png" const imageData = await fetch(imageUrl) .then(response => response.arrayBuffer()) .then(buffer => Buffer.from(buffer).toString('base64')); const o = streamObject({ schema: z.object({ text: z.string(), companyName: z.string().optional().nullable(), }), messages: [ { role: "user", content: [ { type: "text", text: "Describe this image briefly" }, { type: "image", image: imageData, mimeType: "image/png" } ] } ], model: openai("gpt-4.1-nano"), experimental_telemetry: { isEnabled: true, tracer: getTracer() } }); for await (const chunk of o.fullStream) { console.log(chunk); } await Laminar.shutdown(); }; handle().then((r) => { console.log(r); }); ``` ## Related Issues Fixes vercel#6210
## Background The Codex VM uses pnpm 10. I am running into build issues with Codex and want to rule out that they are caused by pnpm version differences. ## Summary Upgrade pnpm to version 10.
## Background Vercel is adding an API for the v0 model per https://vercel.com/docs/v0/api and an AI SDK provider would make it easier for developers to interact with it. ## Summary Added an initial provider for the Vercel API. ## Verification Added examples across several features and executed end to end manually. Co-authored-by: Walter Korman <shaper@vercel.com>
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @ai-sdk/vercel@0.0.1 ### Patch Changes - 42e37fb: feat (provider/vercel): initial vercel provider Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background This pull request adds support for Anthropic's new Claude v4 models. ## Summary Updated the model ids to include `claude-4-opus-20250514` and `claude-4-sonnet-20250514`. ## Tasks - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root)
# Releases ## @ai-sdk/anthropic@1.2.12 ### Patch Changes - f64f4f0: feat (providers/anthropic): add claude v4 models ## @ai-sdk/google-vertex@2.2.23 ### Patch Changes - Updated dependencies [f64f4f0] - @ai-sdk/anthropic@1.2.12 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary Adds guide for Claude 4 models. ## Tasks - [ ] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [ ] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root)
## Background Fal added support for new Flux Kontext models. ## Summary Adds model ids, example, and updates docs. ## Tasks - [ ] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work We might want to add typed provider options here. Currently just pass through the entire provider options into the body.
## Background Backport of vercel#6567 The assign-team-pr-to-author CI action has been failing reliably with an error as: ``` Run gh pr edit $PULL_REQUEST_URL --add-assignee $AUTHOR_LOGIN failed to run git: fatal: not a git repository (or any of the parent directories): .git ``` ## Summary Claude claims best practices are to check out the repo before running `gh`. While it may have worked without a local checkout in the past, `gh` is typically expected to require the repository to be present locally ahead of time. ## Verification Will need to see how subsequent PRs go. Co-authored-by: Walter Korman <shaper@vercel.com>
# Releases ## @ai-sdk/fal@0.1.12 ### Patch Changes - 2e6e5d3: feat (@ai-sdk/fal): support new Flux Kontext models Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Description Add support for new Claude 4 models in Amazon Bedrock provider: - `anthropic.claude-sonnet-4-20250514-v1:0` - `anthropic.claude-opus-4-20250514-v1:0` ## Changes - Added new model IDs to `BedrockChatModelId` type in `bedrock-chat-settings.ts` - Added changeset for version bump ## Notes These are the latest Claude 4 models available in Amazon Bedrock as of May 2025.
…ercel#6597) ## Background The Google API for reasoning was changed in their latest model, leading to Zod errors. ## Summary Make text optional in thinking chunks. Ignore thinking chunks without text. ## Verification Tested example against google api. ## Future work Expose thinking signature using provider metadata, and explore sending it to google in follow-up requests. ## Related Issues Fixes vercel#6589
# Releases ## @ai-sdk/amazon-bedrock@2.2.10 ### Patch Changes - 05b8324: feat (provider/amazon-bedrock): add Claude 4 model ids (claude-sonnet-4-20250514-v1:0, claude-opus-4-20250514-v1:0) ## @ai-sdk/google@1.2.19 ### Patch Changes - f262012: fix (provider/google): prevent error when thinking signature is used ## @ai-sdk/google-vertex@2.2.24 ### Patch Changes - Updated dependencies [f262012] - @ai-sdk/google@1.2.19 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background Some APIs changed and v5 site is launched. ## Summary Updated continueUntil to stopWhen and added reference to new v5 site. ## Tasks - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root)
…NaN with openai-compatiable-provider
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
return actual usage when steaming instead of NaN with openai-compatiable-provider
Background
In my project, used
createOpenAICompatible
andstreamText
.For use actual usage, we used
providerOptions
instreamText
like this:But when I upgrade
@ai-sdk/openai-compatible
package to latest version0.2.14
, I found useage return NaN.Finally, I found some code chang in

0.2.12
, like this:stream_options been overridden, my
providerOptions
about this param never work.I think it's purpose is convenient for get actual usage in this code change, but it's only work in
xai-providers
, becauseincludeUsage
param only exist inxai-provider.ts
in version 0.2.12.Summary
use
to stop overridden