Azure OpenAI provider #1675

NitinS87 · 2024-05-22T19:29:43Z

Feature Description

I just saw the power of new ai/rsc SDK and quite impressed with it. But it's not supported or there is no official documentation for Azure OpenAI integration with it. Will be a huge help if it is supported by it as in Azure OpenAI, users have more control over their data.

Use Case

Implementing a RAGBot with Azure OpenAI.

Additional context

No response

bneigher · 2024-05-23T04:07:47Z

I am also having trouble switching to the ai/rsc SDK (specifically render to streamUI) because I can't figure out how to port the OpenAI constructor over to createOpenAI. It seems like the provider code @ai-sdk/openai is not generating the correct endpoint / parameters.

was:

import {
  render
} from 'ai/rsc'
import OpenAI from 'openai'

export const openai = new OpenAI({
  apiKey: process.env.AZURE_OPEN_AI_KEY,
  baseURL: `${process.env.AZURE_OPEN_AI_ENDPOINT}/openai/deployments/${process.env.AZURE_OPEN_AI_DEPLOYMENT}`,
  defaultQuery: { 'api-version': process.env.AZURE_OPEN_AI_API_VERSION },
  defaultHeaders: { 'api-key': process.env.AZURE_OPEN_AI_KEY }
})

async function submitUserMessage(content: string) {
  'use server'

  const ui = render({
    model: 'gpt-3.5-turbo',
    provider: openai,
    ...
  })
}

now:

import {
  streamUI
} from 'ai/rsc';
import { createOpenAI } from '@ai-sdk/openai';

const openAI = createOpenAI({
  apiKey: process.env.AZURE_OPEN_AI_KEY,
  baseURL: `${process.env.AZURE_OPEN_AI_ENDPOINT}/openai/deployments/${process.env.AZURE_OPEN_AI_DEPLOYMENT}`,
  headers: {
    'api-version': process.env.AZURE_OPEN_AI_API_VERSION
  },
  compatibility: 'strict'
})

async function submitUserMessage(content: string) {
  'use server'

  const ui = await streamUI({
      model: openAI(process.env.AZURE_OPEN_AI_DEPLOYMENT),
      initial: <SpinnerMessage />,
      ...
  })
  ...
}

lgrammel · 2024-05-23T12:41:46Z

Azure OpenAI is not fully compatible with OpenAI and will need a separate provider

patrick-moore · 2024-05-23T18:30:39Z

I was looking into this a little bit yesterday. I set the base url to my azure deployment, added the api-key header, and then modified the openai provider endpoints to include an api-version query parameter. This worked for the first request, but not subsequent ones that included chat history. I didn't have a chance to look in detail but I believe it was related to the openai provider including image type when I was using a model not supporting image type

patrick-moore · 2024-05-25T06:26:10Z

Looked into this some more, and there is some funkiness on the Azure API. Sending a single message with the {role: "user", content: [{type: "text", text: string}]} returns a 200:

But, once you add multiple messages you get a misleading 500 error:

I was able to get chat streaming working by modifying the OpenAI provider to accept some Azure specific values and format user messages as {role: "user", content: string}. Once I get time to test out the other functionality I can push this up as a community provider

microflyer · 2024-05-27T01:51:26Z

I spent hours trying to get it to work with Azure Open AI, but unfortunately, I couldn't make it happen. I'll just have to wait for the official Azure Open AI provider now.

patrick-moore · 2024-05-28T21:50:53Z

I'm hoping to get something up later this week, and will update this thread when I do. Pretty sure I have everything working for text inputs with ai/rsc.

The other change I've needed so far is adjusting the openaiChatChunkSchema to allow for an empty object property when streaming. It seems like Azure will send an empty first chunk sometimes, that can be ignored as far as I can tell.

miguelvictor · 2024-05-30T14:41:59Z

we also need this because our company is Azure only 🥲

patrick-moore · 2024-05-30T16:22:07Z

I'll have this up today for chat models and text input. Just trying to figure out some Github action stuff.

patrick-moore · 2024-05-30T19:39:32Z

https://github.com/patrickrmoore/azure-openai-provider

This has had minimal testing and is very much use at your own risk at the moment

lgrammel · 2024-06-01T07:38:58Z

@patrick-moore heads up that we will prob integrate azure-openai as first party in the repo

patrick-moore · 2024-06-01T15:52:07Z

Awesome, I think that makes the most sense. The changes were pretty minimal and my current package is not set up to make integrating upstream changes easy. Adding a few extension points to the existing OpenAI provider would solve that. I'll hold off on putting in a PR to list this as a community provider. Let me know if yall would like any help on this!

yangcheng · 2024-06-04T03:50:35Z

@patrick-moore thanks so much for your work! unblocked me today

yangcheng · 2024-06-04T03:51:25Z

@patrick-moore heads up that we will prob integrate azure-openai as first party in the repo

@lgrammel is there a timeline for azure-openai first party provider? thanks

simpel · 2024-06-04T08:08:35Z

+1 on the timeline! @lgrammel :) Goes without saying but stellar work in this SDK!!

lgrammel · 2024-06-04T08:33:17Z

I'm blocked (do not have Azure OpenAI access) so I cannot provide a timeline

lgrammel · 2024-06-11T04:44:46Z

#1901

lgrammel · 2024-06-11T07:33:26Z

https://www.npmjs.com/package/@ai-sdk/azure

wong2 · 2024-06-11T08:38:17Z

I tried it immediately, but it seems there's a problem.

Cannot find module '[hidden]/node_modules/.pnpm/@ai-sdk+azure@0.0.1_zod@3.23.8/node_modules/@ai-sdk/openai/internal/dist/index.mjs' imported from [hidden]/node_modules/.pnpm/@ai-sdk+azure@0.0.1_zod@3.23.8/node_modules/@ai-sdk/azure/dist/index.mjs

I've already upgraded @ai-sdk/openai to 0.0.25, but it doesn't export the internal module.

lgrammel · 2024-06-11T08:53:05Z

@wong2 thanks for reporting. #1912

lgrammel · 2024-06-11T09:20:38Z

I tried it immediately, but it seems there's a problem.
Cannot find module '[hidden]/node_modules/.pnpm/@ai-sdk+azure@0.0.1_zod@3.23.8/node_modules/@ai-sdk/openai/internal/dist/index.mjs' imported from [hidden]/node_modules/.pnpm/@ai-sdk+azure@0.0.1_zod@3.23.8/node_modules/@ai-sdk/azure/dist/index.mjs
I've already upgraded @ai-sdk/openai to 0.0.25, but it doesn't export the internal module.

https://github.com/vercel/ai/releases/tag/%40ai-sdk/azure%400.0.2

wong2 · 2024-06-11T09:26:21Z

New error:

Type validation failed: Value: {"choices":[{"content_filter_offsets":{"check_offset":159,"start_offset":159,"end_offset":300},"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"finish_reason":null,"index":0}],"created":0,"id":"","model":"","object":""}.
Error message: [
  {
    "code": "invalid_union",
    "unionErrors": [
      {
        "issues": [
          {
            "code": "invalid_type",
            "expected": "object",
            "received": "undefined",
            "path": [
              "choices",
              0,
              "delta"
            ],
            "message": "Required"
          }
        ],
        "name": "ZodError"
      },
      {
        "issues": [
          {
            "code": "invalid_type",
            "expected": "object",
            "received": "undefined",
            "path": [
              "error"
            ],
            "message": "Required"
          }
        ],
        "name": "ZodError"
      }
    ],
    "path": [],
    "message": "Invalid input"
  }
]

lgrammel · 2024-06-11T09:44:41Z

@wong2 what inputs / model are you using?

wong2 · 2024-06-11T09:52:35Z

@lgrammel GPT-4o, the input is just hi

lgrammel · 2024-06-11T11:38:37Z

@wong2 i was not able to reproduce this with our deployments. However, I've prepared a PR that makes delta optional: #1915

lgrammel · 2024-06-11T12:02:36Z

@wong2 can you give https://github.com/vercel/ai/releases/tag/%40ai-sdk/azure%400.0.3 a try when you get a chance? i was not able to reproduce the issue, but this should add more robustness to the stream chunk validation

wong2 · 2024-06-11T12:06:46Z

It works now. Thanks.

bneigher · 2024-06-11T18:06:58Z

@lgrammel awesome, giving this a spin now for streamUI. However it's unclear how I pass in the four necessary configs:

import {
  streamUI
} from 'ai/rsc';
import { createAzure } from '@ai-sdk/azure';

const azure = createAzure({
    apiKey: process.env.AZURE_OPEN_AI_KEY,
    resourceName: process.env.AZURE_OPEN_AI_DEPLOYMENT
})(process.env.AZURE_OPEN_AI_API_VERSION)

async function submitUserMessage(content: string) {
  'use server'

  const ui = await streamUI({
      model: azure,
      initial: <SpinnerMessage />,
      ...
  })
  ...
}

It looks like the interface here is only resourceName and apiKey

But I need to also set a baseURL and an api version, no?

AZURE_OPEN_AI_KEY=$$$$$$
AZURE_OPEN_AI_ENDPOINT=yyyyy.zzzz.openai.azure.com/
AZURE_OPEN_AI_DEPLOYMENT=xxxxx-xxxx-gpt-4o
AZURE_OPEN_AI_API_VERSION=2024-05-01-preview

example:

// ?
createAzure({
    apiKey: process.env.AZURE_OPEN_AI_KEY,
    resourceName: process.env.AZURE_OPEN_AI_DEPLOYMENT
})(process.env.AZURE_OPEN_AI_API_VERSION)

// working
createOpenAI({
    apiKey: process.env.OPEN_AI_KEY,
    baseURL: process.env.OPEN_AI_BASE_URL,
    organization: 'org-xxxxx',
    headers: { ... }
})('gpt-3.5-turbo')

let me know how I can use this!

lgrammel · 2024-06-11T18:23:08Z

@bneigher

You can use:

import { createAzure } from '@ai-sdk/azure';

const azure = createAzure({
  resourceName: 'your-resource-name', // Azure resource name
  apiKey: 'your-api-key',
});

and then create a model:

const model = azure('your-deployment-name');

here are the docs: https://sdk.vercel.ai/providers/ai-sdk-providers/azure

and if it helps, here is how the url is assembled internally: https://github.com/vercel/ai/blob/main/packages/azure/src/azure-openai-provider.ts#L64

bneigher · 2024-06-11T19:29:51Z

@lgrammel I got it, yea that looks right and populates the endpoint in https://github.com/vercel/ai/blob/main/packages/azure/src/azure-openai-provider.ts#L64.

But it still seems like something is missing, it doesn't seem to send anything out with streamUI. I am using this sample app for implementation https://github.com/vercel/ai-chatbot/tree/main. Going to dig a bit more into it..

Works great!

miguelvictor · 2024-06-12T08:24:02Z

I'm noticing that streaming with azure is clunky than the one with openai but maybe this is just an azure problem 🤔

lgrammel · 2024-06-12T08:28:50Z

@miguelvictor yes that's a know azure issue

miguelvictor · 2024-06-12T17:08:49Z

I'm also noticing that the same tool is being called twice (at the same time; resulting to Error: .update(): UI stream is already closed. errors) when using the azure provider but the problem goes away when I switch back to the openai provider.

lgrammel · 2024-06-12T17:38:00Z

@miguelvictor do you have a test case that i could use to reproduce this?

miguelvictor · 2024-06-12T17:44:47Z

Sorry, it also happened with the openai provider. Is it possible to only process one tool per streamUI call?

lgrammel · 2024-06-12T17:56:58Z

@miguelvictor not yet but I plan to implement #1893 which should help

miguelvictor · 2024-06-12T17:58:19Z

Cool! Thank you so much!

miguelvictor · 2024-06-13T10:31:15Z

Hi @lgrammel! I see you have already merged #1893 and you also added code for the azure provider. Did you perhaps forget to release the @ai-sdk/azure provider package?

lgrammel · 2024-06-13T11:14:51Z

@miguelvictor it is automatically released when the openai package changes: https://github.com/vercel/ai/releases/tag/%40ai-sdk%2Fazure%400.0.4 (should include that setting)

miguelvictor · 2024-06-13T12:06:36Z

Sorry, bun, for some reason, didn't pickup the semver update 🥲
I installed it manually but im getting another error: AI_APICallError: Unknown parameter: 'parallel_tool_calls'..

I'm using it like this:

import { createAzure } from "@ai-sdk/azure"

export const azure = createAzure({
  resourceName: env.AZURE_OPENAI_RESOURCE_NAME,
  apiKey: env.AZURE_OPENAI_KEY,
})(env.AZURE_OPENAI_DEPLOYMENT_CHAT, { parallelToolCalls: false })

lgrammel · 2024-06-13T13:18:29Z

@miguelvictor That means that the Azure backend does not support it yet I think

lgrammel self-assigned this May 23, 2024

lgrammel added ai/core ai/provider labels May 23, 2024

lgrammel changed the title ~~Azure OpenAI integration with ai/rsc~~ Azure OpenAI provider May 24, 2024

lgrammel added the enhancement New feature or request label May 24, 2024

This was referenced May 24, 2024

Azure OpenAI support #1643

Closed

Inconsistence tools call response with Azure Openai API #1369

Closed

Azure OpenAI function calling does not work #1156

Closed

Streaming Text Response doesn't work with Azure Open AI (ai: "^2.2.35") #978

Closed

lgrammel removed the ai/core label Jun 5, 2024

lgrammel closed this as completed Jun 11, 2024

Azure OpenAI provider #1675

Azure OpenAI provider #1675

Comments

NitinS87 commented May 22, 2024

Feature Description

Use Case

Additional context

bneigher commented May 23, 2024

was:

now:

lgrammel commented May 23, 2024

patrick-moore commented May 23, 2024

patrick-moore commented May 25, 2024 • edited Loading

microflyer commented May 27, 2024

patrick-moore commented May 28, 2024 • edited Loading

miguelvictor commented May 30, 2024

patrick-moore commented May 30, 2024

patrick-moore commented May 30, 2024

lgrammel commented Jun 1, 2024

patrick-moore commented Jun 1, 2024 • edited Loading

yangcheng commented Jun 4, 2024

yangcheng commented Jun 4, 2024

simpel commented Jun 4, 2024

lgrammel commented Jun 4, 2024

lgrammel commented Jun 11, 2024

lgrammel commented Jun 11, 2024

wong2 commented Jun 11, 2024 • edited Loading

lgrammel commented Jun 11, 2024

lgrammel commented Jun 11, 2024

wong2 commented Jun 11, 2024

lgrammel commented Jun 11, 2024

wong2 commented Jun 11, 2024

lgrammel commented Jun 11, 2024

lgrammel commented Jun 11, 2024

wong2 commented Jun 11, 2024

bneigher commented Jun 11, 2024 • edited Loading

lgrammel commented Jun 11, 2024 • edited Loading

bneigher commented Jun 11, 2024 • edited Loading

miguelvictor commented Jun 12, 2024

lgrammel commented Jun 12, 2024

miguelvictor commented Jun 12, 2024

lgrammel commented Jun 12, 2024

miguelvictor commented Jun 12, 2024

lgrammel commented Jun 12, 2024 • edited Loading

miguelvictor commented Jun 12, 2024

miguelvictor commented Jun 13, 2024

lgrammel commented Jun 13, 2024

miguelvictor commented Jun 13, 2024 • edited Loading

lgrammel commented Jun 13, 2024

patrick-moore commented May 25, 2024 •

edited

Loading

patrick-moore commented May 28, 2024 •

edited

Loading

patrick-moore commented Jun 1, 2024 •

edited

Loading

wong2 commented Jun 11, 2024 •

edited

Loading

bneigher commented Jun 11, 2024 •

edited

Loading

lgrammel commented Jun 11, 2024 •

edited

Loading

bneigher commented Jun 11, 2024 •

edited

Loading

lgrammel commented Jun 12, 2024 •

edited

Loading

miguelvictor commented Jun 13, 2024 •

edited

Loading