feat(openai-compatible): add support for file and audio content parts #6369
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…nge (vercel#5531) Co-authored-by: Carl Brugger <cebrugg@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…#5541) Co-authored-by: Bram Meerten <bram.meerten@acagroup.be>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background This pull request fixes a docs formatting issue due to a missing comma in useChat reference docs. ## Summary Fixed formatting in useChat reference docs by adding a comma. ## Tasks - [x] Documentation has been added / updated (for bug fixes / features) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Related Issues vercel#6052
## Background The Dify provider allows developers to easily integrate Dify's application workflow with their applications using the Vercel AI SDK. ## Summary This PR adds the Dify provider documentation to the AI SDK, enabling users to understand how to implement and use the Dify integration. ## Tasks - [x] Documentation has been added for the Dify provider - [x] Installation instructions are included - [x] Usage examples for both cloud and self-hosted instances are provided ## Related Issues [Support Dify Provider](vercel#4084) --------- Co-authored-by: Lars Grammel <lars.grammel@gmail.com>
<!-- Welcome to contributing to AI SDK! We're excited to see your changes. We suggest you read the following contributing guide we've created before submitting: https://github.com/vercel/ai/blob/main/CONTRIBUTING.md --> ## Background <!-- Why was this change necessary? --> ## Summary <!-- What did you change? --> ## Verification <!-- For features & bugfixes. Please explain how you *manually* verified that the change works end-to-end as expected (independent of automated tests). Remove the section if it's not needed (e.g. for docs). --> ## Tasks <!-- This task list is intended to help you keep track of what you need to do. Feel free to add tasks and remove unnecessary tasks as needed. Please check if the PR fulfills the following requirements: --> - [ ] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [ ] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work <!-- Feel free to mention things not covered by this PR that can be done in future PRs. Remove the section if it's not needed. --> ## Related Issues <!-- List related issues here, e.g. "Fixes vercel#1234". Remove the section if it's not needed. -->
## Background The safetyRating was an empty object for some reason ## Summary Fixes vercel#6109
…el#6133) ## Background Before in issues like vercel#5499 & vercel#5833, if users ran into an issue where the messages array failed to parse correctly they'd run into a difficult to debug error `Invalid prompt: messages must be an array of CoreMessage or UIMessage` ## Summary This PR extends these error messages to include more details on what exactly went wrong, which is useful information for debugging when stuff breaks. On security: I think this is an okay change since the data sent is not private
This adds gemini-2.5-pro-preview-05-06 to AI SDK
## Background Laminar has updated its SDK and docs and added an example Next.js app. This is needed to better guide AI SDK users ## Summary A guide on Laminar as an observability provider is updated.
## Background The below MCP SSE message will make the stream reader timeout. ``` data: /message?sessionId=123\r\nevent: endpoint ``` ## Summary Fix the `splitLines` parser issue. The CRLF pointer was incorrectly implemented.
# Releases ## ai@4.3.15 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/react@1.2.12 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/amazon-bedrock@2.2.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/anthropic@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/azure@1.3.23 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai@1.3.22 ## @ai-sdk/cerebras@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/cohere@1.2.10 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/deepinfra@0.2.15 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/deepseek@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/elevenlabs@0.0.3 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/fal@0.1.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/fireworks@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/google@1.2.17 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/google-vertex@2.2.21 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/anthropic@1.2.11 - @ai-sdk/google@1.2.17 ## @ai-sdk/groq@1.2.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/hume@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/lmnt@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/luma@0.1.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/mistral@1.2.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/openai@1.3.22 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/openai-compatible@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/perplexity@1.1.9 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/provider-utils@2.2.8 ### Patch Changes - d87b9d1: fix(provider-utils): fix SSE parser bug (CRLF) ## @ai-sdk/react@1.2.12 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/replicate@0.2.8 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/revai@0.0.2 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/solid@1.2.13 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/svelte@2.1.12 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/togetherai@0.2.14 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 ## @ai-sdk/ui-utils@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 ## @ai-sdk/valibot@0.1.27 ### Patch Changes - ai@4.3.15 ## @ai-sdk/vue@1.2.11 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/ui-utils@1.2.11 ## @ai-sdk/xai@1.2.16 ### Patch Changes - Updated dependencies [d87b9d1] - @ai-sdk/provider-utils@2.2.8 - @ai-sdk/openai-compatible@0.2.14 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ming stream (vercel#6224) ## Background This pull request updates the documentation for the GET handler in the resumable streams section to account for a race condition that may occur when the stream ends in-flight. ## Summary Updated the route by adding the code snippet that handles the above condition. ## Tasks - [x] Documentation has been added / updated (for bug fixes / features) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Related Issues vercel#6052
## Background Bug with Vue that led to status not updating when tab was changed. ## Summary Changes status from using SWR to using Vue ref.
# Releases ## @ai-sdk/vue@1.2.12 ### Patch Changes - b54cb59: fix (ai-sdk/vue): fix status reactivity Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Background In mcp doc, `mcpClient.tools()` need to await ## Verification Just fix doc problem Co-authored-by: georgezouq <george@staringos.com> Co-authored-by: Lars Grammel <lars.grammel@gmail.com>
## Background [Vertex now supports extraction of thinking tokens in certain Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking). When the configuration is passed via `providerOptions`, the sdk: 1. Did not extract reasoning tokens 2. Did not pass `include_thoughts` to the provider ## Summary Added extraction logic to google-generative-ai package to parse reasoning tokens. Added a `includeThoughts` switch to the `thinkingConfig` for vertex models. ## Verification I verified it manually. Testable via examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily copiable to google provider. ## Related Issues Fixes vercel#6259
## Background Release requires change to `google` provider package. ## Summary Add changeset.
# Releases ## @ai-sdk/google@1.2.18 ### Patch Changes - 4b2e1b0: Add reasoning token output support for gemini models via Vertex AI Provider ## @ai-sdk/google-vertex@2.2.22 ### Patch Changes - fe24216: Add reasoning token output support for gemini models via Vertex AI Provider - Updated dependencies [4b2e1b0] - @ai-sdk/google@1.2.18 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
) ## Background Like many other OpenTelemetry instrumentations, Laminar depends on `@opentelemetry/instrumentation`, which, in turn, depends on `require-in-the-middle` and `import-in-the-middle`. Importing and initializing Laminar inside Next.js `instrumentation.ts` file causes Next.js to try resolving these two packages, but fails, and results in: - Laminar not being able to send traces. This is because unlike many other instrumentation libraries, Laminar is not intrusive and does not set its tracer provider globally (so that others, e.g. `@vercel/otel` can set theirs). - Error messages (see below) We have tried many different things to debug, including bundling Laminar differently, shipping those two packages within Laminar as `noExternal`, adding a separate entrypoint in our package for Next.js, but nothing seems to have worked. The only thing that's worked was adding `@lmnr-ai/lmnr` in `serverExternalPackages` in [next.config](https://nextjs.org/docs/app/api-reference/config/next-config-js/serverExternalPackages). ## Summary Add a subsection within the Next.js section that describes
…#6315) ## Background It is not clear what AI SDK version user-submitted issues refer to. ## Summary Add version section to issue template.
## Background Integrating with Patronus! ## Summary A new Patronus.mdx file showing how logs can be imported into Patronus via OTel. ## Tasks - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) --------- Co-authored-by: Snigdha Banda <snigdhabanda@Snigdhas-MacBook-Pro.local> Co-authored-by: nicoalbanese <gcalbanese96@gmail.com> Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com>
## Summary Announce v5 Alpha ## Tasks - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root)
Collaborator
|
Sorry, we didn't get to this in time. Please re-create the PR if the change is still needed and we'll take a look! This comment is automated as part of our triage process. Truly sorry for the inconvenieince, we are doing our best 🖤 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Currently, the
openai-compatibleprovider does not supportfileoraudiocontent parts, leading to errors when sending files (e.g., PDFs) or audio inputs. This limits compatibility with OpenAI-compatible APIs and prevents use cases like document or audio processing.Summary
filecontent parts (e.g., PDF files) using Base64 encoding.audio/mp3,audio/mpeg, andaudio/wavcontent parts asinput_audio.ChatCompletionContentPartFileandChatCompletionContentPartInputAudio.Verification
UnsupportedFunctionalityError.Tasks
pnpm changesetin the project root)pnpm prettier-fixin the project root)Future Work
Related Issues
N/A