Skip to content

fix(openai-compatible): return actual usage when steaming instead of NaN #6648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 197 commits into
base: main
Choose a base branch
from

Conversation

ki7ick
Copy link

@ki7ick ki7ick commented Jun 6, 2025

return actual usage when steaming instead of NaN with openai-compatiable-provider

Background

In my project, used createOpenAICompatible and streamText.

For use actual usage, we used providerOptions in streamText like this:

streamText({
  ...,
  providerOptions: {
    'xxxprovider': {
      stream_options: { include_usage: true }
    },
  },
  ...
})

But when I upgrade @ai-sdk/openai-compatible package to latest version 0.2.14, I found useage return NaN.

Finally, I found some code chang in 0.2.12, like this:
image

stream_options been overridden, my providerOptions about this param never work.

I think it's purpose is convenient for get actual usage in this code change, but it's only work in xai-providers, because includeUsage param only exist in xai-provider.ts in version 0.2.12.

Summary

use

...this.config.includeUsage && {
  include_usage: true 
}

to stop overridden

dylanmoz and others added 30 commits April 3, 2025 08:10
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…nge (vercel#5531)

Co-authored-by: Carl Brugger <cebrugg@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…#5541)

Co-authored-by: Bram Meerten <bram.meerten@acagroup.be>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
)

Co-authored-by: Sam Denty <sam@samdenty.com>
Co-authored-by: Sam Denty <samddenty@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions bot and others added 29 commits May 7, 2025 14:12
# Releases
## ai@4.3.15

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/react@1.2.12
    -   @ai-sdk/ui-utils@1.2.11

## @ai-sdk/amazon-bedrock@2.2.9

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/anthropic@1.2.11

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/azure@1.3.23

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai@1.3.22

## @ai-sdk/cerebras@0.2.14

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

## @ai-sdk/cohere@1.2.10

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/deepinfra@0.2.15

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

## @ai-sdk/deepseek@0.2.14

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

## @ai-sdk/elevenlabs@0.0.3

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/fal@0.1.11

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/fireworks@0.2.14

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

## @ai-sdk/google@1.2.17

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/google-vertex@2.2.21

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/anthropic@1.2.11
    -   @ai-sdk/google@1.2.17

## @ai-sdk/groq@1.2.9

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/hume@0.0.2

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/lmnt@0.0.2

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/luma@0.1.8

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/mistral@1.2.8

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/openai@1.3.22

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/openai-compatible@0.2.14

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/perplexity@1.1.9

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/provider-utils@2.2.8

### Patch Changes

-   d87b9d1: fix(provider-utils): fix SSE parser bug (CRLF)

## @ai-sdk/react@1.2.12

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/ui-utils@1.2.11

## @ai-sdk/replicate@0.2.8

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/revai@0.0.2

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/solid@1.2.13

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/ui-utils@1.2.11

## @ai-sdk/svelte@2.1.12

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/ui-utils@1.2.11

## @ai-sdk/togetherai@0.2.14

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

## @ai-sdk/ui-utils@1.2.11

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8

## @ai-sdk/valibot@0.1.27

### Patch Changes

-   ai@4.3.15

## @ai-sdk/vue@1.2.11

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/ui-utils@1.2.11

## @ai-sdk/xai@1.2.16

### Patch Changes

-   Updated dependencies [d87b9d1]
    -   @ai-sdk/provider-utils@2.2.8
    -   @ai-sdk/openai-compatible@0.2.14

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ming stream (vercel#6224)

## Background

This pull request updates the documentation for the GET handler in the
resumable streams section to account for a race condition that may occur
when the stream ends in-flight.

## Summary

Updated the route by adding the code snippet that handles the above
condition.

## Tasks

- [x] Documentation has been added / updated (for bug fixes / features)
- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)

## Related Issues

vercel#6052
## Background

Bug with Vue that led to status not updating when tab was changed.

## Summary

Changes status from using SWR to using Vue ref.
# Releases
## @ai-sdk/vue@1.2.12

### Patch Changes

-   b54cb59: fix (ai-sdk/vue): fix status reactivity

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background

In mcp doc, `mcpClient.tools()` need to await


## Verification

Just fix doc problem

Co-authored-by: georgezouq <george@staringos.com>
Co-authored-by: Lars Grammel <lars.grammel@gmail.com>
## Background

[Vertex now supports extraction of thinking tokens in certain Gemini
models](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking).

When the configuration is passed via `providerOptions`, the sdk:
1. Did not extract reasoning tokens
2. Did not pass `include_thoughts` to the provider

## Summary

Added extraction logic to google-generative-ai package to parse
reasoning tokens.

Added a `includeThoughts` switch to the `thinkingConfig` for vertex
models.

## Verification

I verified it manually. Testable via
examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily
copiable to google provider.

## Related Issues
Fixes vercel#6259
## Background

Release requires change to `google` provider package.

## Summary

Add changeset.
# Releases
## @ai-sdk/google@1.2.18

### Patch Changes

- 4b2e1b0: Add reasoning token output support for gemini models via
Vertex AI Provider

## @ai-sdk/google-vertex@2.2.22

### Patch Changes

- fe24216: Add reasoning token output support for gemini models via
Vertex AI Provider
-   Updated dependencies [4b2e1b0]
    -   @ai-sdk/google@1.2.18

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
)

## Background

Like many other OpenTelemetry instrumentations, Laminar depends on
`@opentelemetry/instrumentation`, which, in turn, depends on
`require-in-the-middle` and `import-in-the-middle`. Importing and
initializing Laminar inside Next.js `instrumentation.ts` file causes
Next.js to try resolving these two packages, but fails, and results in:

- Laminar not being able to send traces. This is because unlike many
other instrumentation libraries, Laminar is not intrusive and does not
set its tracer provider globally (so that others, e.g. `@vercel/otel`
can set theirs).
- Error messages (see below)

We have tried many different things to debug, including bundling Laminar
differently, shipping those two packages within Laminar as `noExternal`,
adding a separate entrypoint in our package for Next.js, but nothing
seems to have worked.

The only thing that's worked was adding `@lmnr-ai/lmnr` in
`serverExternalPackages` in
[next.config](https://nextjs.org/docs/app/api-reference/config/next-config-js/serverExternalPackages).

## Summary

Add a subsection within the Next.js section that describes
…#6315)

## Background

It is not clear what AI SDK version user-submitted issues refer to.

## Summary

Add version section to issue template.
## Background

Integrating with Patronus!

## Summary
A new Patronus.mdx file showing how logs can be imported into Patronus
via OTel.

## Tasks

- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)

---------

Co-authored-by: Snigdha Banda <snigdhabanda@Snigdhas-MacBook-Pro.local>
Co-authored-by: nicoalbanese <gcalbanese96@gmail.com>
Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com>
## Summary

Announce v5 Alpha

## Tasks
- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)
…spans (vercel#6357)

## Background

`generateObject`, `generateText`, `streamText`, and `streamObject`
currently call `JSON.stringify` on the input messages. If the input
messages contain an image, it is most likely normalized into a
`Uint8Array`.

`JSON.stringify` does not the most obvious things to TypedArrays
including `Uint8Array`.

```javascript
// this returns '{"0": 1,"1": 2,"2": 3}', where I'd expect this to be '[1,2,3]'
JSON.stringify(new Uint8array([1, 2, 3]))
```

In practice, this results in bloating images by about 5-15x depending on
the original image size. For Laminar, for example, a span with 3 avg
sized images will not be able to be sent as it is larger than the
(reasonably high) gRPC payload size for our traces endpoint.

From [MDN
docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#examples):
```javascript
// TypedArray
JSON.stringify([new Int8Array([1]), new Int16Array([1]), new Int32Array([1])]);
// '[{"0":1},{"0":1},{"0":1}]'
JSON.stringify([
  new Uint8Array([1]),
  new Uint8ClampedArray([1]),
  new Uint16Array([1]),
  new Uint32Array([1]),
]);
// '[{"0":1},{"0":1},{"0":1},{"0":1}]'
JSON.stringify([new Float32Array([1]), new Float64Array([1])]);
// '[{"0":1},{"0":1}]'
```

## Summary

Added a function that maps over messages in a `LanguageModelV1Prompt`
and maps over content parts in each message, replacing `UInt8Array`s
with raw base64 strings instead.

Call this function when calling `recordSpan` for the inner
(doStream/doGenerate) span in `generateObject`, `generateText`,
`streamText`, and `streamObject`.

## Verification

Ran this small script against a local instance of Laminar and logged the
Telemetry payloads (span attributes) on the backend to verify that they
are indeed base64.

```javascript
import { Laminar, getTracer } from '@lmnr-ai/lmnr'

Laminar.initialize();

import { openai } from '@ai-sdk/openai'
import { generateText, generateObject, streamText, streamObject, tool } from "ai";
import { z } from "zod";
import dotenv from "dotenv";

dotenv.config();

const handle = async () => {
  const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/b/bc/CoinEx.png"
  const imageData = await fetch(imageUrl)
    .then(response => response.arrayBuffer())
    .then(buffer => Buffer.from(buffer).toString('base64'));

  const o = streamObject({
    schema: z.object({
      text: z.string(),
      companyName: z.string().optional().nullable(),
    }),
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this image briefly"
          },
          {
            type: "image",
            image: imageData,
            mimeType: "image/png"
          }
        ]
      }
    ],
    model: openai("gpt-4.1-nano"),
    experimental_telemetry: {
      isEnabled: true,
      tracer: getTracer()
    }
  });

  for await (const chunk of o.fullStream) {
    console.log(chunk);
  }
  await Laminar.shutdown();
};

handle().then((r) => {
    console.log(r);
});
```

## Related Issues

Fixes vercel#6210
# Releases
## ai@4.3.16

### Patch Changes

-   ed0ebeb: Avoid JSON.strinfigy on UInt8Arrays for telemetry

## @ai-sdk/valibot@0.1.28

### Patch Changes

-   Updated dependencies [ed0ebeb]
    -   ai@4.3.16

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background

The Codex VM uses pnpm 10. I am running into build issues with Codex and
want to rule out that they are caused by pnpm version differences.

## Summary

Upgrade pnpm to version 10.
## Background

Vercel is adding an API for the v0 model per
https://vercel.com/docs/v0/api and an AI SDK provider would make it
easier for developers to interact with it.

## Summary

Added an initial provider for the Vercel API.

## Verification

Added examples across several features and executed end to end manually.

Co-authored-by: Walter Korman <shaper@vercel.com>
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to main, this PR will
be updated.


# Releases
## @ai-sdk/vercel@0.0.1

### Patch Changes

-   42e37fb: feat (provider/vercel): initial vercel provider

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background

This pull request adds support for Anthropic's new Claude v4 models.

## Summary

Updated the model ids to include `claude-4-opus-20250514` and
`claude-4-sonnet-20250514`.

## Tasks

- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)
# Releases
## @ai-sdk/anthropic@1.2.12

### Patch Changes

-   f64f4f0: feat (providers/anthropic): add claude v4 models

## @ai-sdk/google-vertex@2.2.23

### Patch Changes

-   Updated dependencies [f64f4f0]
    -   @ai-sdk/anthropic@1.2.12

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary

Adds guide for Claude 4 models.

## Tasks

- [ ] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [ ] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)
## Background

Fal added support for new Flux Kontext models.

## Summary

Adds model ids, example, and updates docs.

## Tasks

- [ ] Tests have been added / updated (for bug fixes / features)
- [x] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)

## Future Work

We might want to add typed provider options here. Currently just pass
through the entire provider options into the body.
## Background

Backport of vercel#6567

The assign-team-pr-to-author CI action has been failing reliably with an
error as:

```
Run gh pr edit $PULL_REQUEST_URL --add-assignee $AUTHOR_LOGIN
failed to run git: fatal: not a git repository (or any of the parent directories): .git
```

## Summary

Claude claims best practices are to check out the repo before running
`gh`. While it may have worked without a local checkout in the past,
`gh` is typically expected to require the repository to be present
locally ahead of time.

## Verification

Will need to see how subsequent PRs go.

Co-authored-by: Walter Korman <shaper@vercel.com>
# Releases
## @ai-sdk/fal@0.1.12

### Patch Changes

-   2e6e5d3: feat (@ai-sdk/fal): support new Flux Kontext models

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Description
Add support for new Claude 4 models in Amazon Bedrock provider:
- `anthropic.claude-sonnet-4-20250514-v1:0`
- `anthropic.claude-opus-4-20250514-v1:0`

## Changes
- Added new model IDs to `BedrockChatModelId` type in
`bedrock-chat-settings.ts`
- Added changeset for version bump

## Notes
These are the latest Claude 4 models available in Amazon Bedrock as of
May 2025.
…ercel#6597)

## Background

The Google API for reasoning was changed in their latest model, leading
to Zod errors.

## Summary

Make text optional in thinking chunks. Ignore thinking chunks without
text.

## Verification

Tested example against google api.

## Future work
Expose thinking signature using provider metadata, and explore sending
it to google in follow-up requests.

## Related Issues
Fixes vercel#6589
# Releases
## @ai-sdk/amazon-bedrock@2.2.10

### Patch Changes

- 05b8324: feat (provider/amazon-bedrock): add Claude 4 model ids
(claude-sonnet-4-20250514-v1:0, claude-opus-4-20250514-v1:0)

## @ai-sdk/google@1.2.19

### Patch Changes

- f262012: fix (provider/google): prevent error when thinking signature
is used

## @ai-sdk/google-vertex@2.2.24

### Patch Changes

-   Updated dependencies [f262012]
    -   @ai-sdk/google@1.2.19

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background

Some APIs changed and v5 site is launched.

## Summary

Updated continueUntil to stopWhen and added reference to new v5 site.

## Tasks

- [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the
project root)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.