Fix streaming for function calls

[Zakinator123]

Context

I'm trying to build an LLM agent that operates primarily on the client side that users can interact with from a chat interface.

With OpenAI's new function-calling capabilities, it's up to the model to decide whether or not it will call a function (if functions are provided to it). If using an Edge Runtime route handler similar to the Vercel AI SDK example here, the route handler should be able to support streaming back a function call in addition to normal chat completion responses from the assistant role. Without this PR, Vercel's streaming utils simply return an empty stream if OpenAI responds with a function call.

Testing

I've only manually tested this change so far, but if maintainers would like to move forward with the approach, I can write a unit test as well.

To see how the streamed response chunks from OpenAI are broken up, Postman is a great tool.

Since the Vercel AI SDK does not yet support function-calling in its useChat() hook or any other method, I've got my own custom logic calling the Edge route handler for now that's passing functions in the request (See below). I also am using an updated version of openai-edge (PR here) in order to use function calling in the Next Edge Runtime.

Here's the rudimentary code I hacked together to test this. Note that the code below does not render the stream chunk by chunk on the client side yet (though that would probably be trivial to add).

'use client';

import { useState } from 'react';

// This type comes from the updated version of openai-edge that I copied over into my repo.
import { CreateChatCompletionRequest } from '@/openai-edge-function-calling';

export default function ChatWithFunctionCall() {
  const [input, setInput] = useState('');
  const [response, setResponse] = useState('');

  const functions = [
    {
      'name': 'get_current_weather',
      'description': 'Get the current weather',
      'parameters': {
        'type': 'object',
        'properties': {
          'location': {
            'type': 'string',
            'description': 'The city and state, e.g. San Francisco, CA'
          },
          'format': {
            'type': 'string',
            'enum': ['celsius', 'fahrenheit'],
            'description': 'The temperature unit to use. Infer this from the users location.'
          }
        },
        'required': ['location', 'format']
      }
    }];

  const decoder = new TextDecoder();

  function decodeAIStreamChunk(chunk: Uint8Array): string {
    return decoder.decode(chunk);
  }

  const handleSubmit = async (e: any) => {
    e.preventDefault()

    const createChatCompletionRequest: CreateChatCompletionRequest = {
      model: 'gpt-3.5-turbo',
      messages: [{ role: 'user', content: input }],
      functions: functions
    };

    const res = await fetch('/api/chat/', {
      method: 'POST',
      body: JSON.stringify(createChatCompletionRequest)
    });

    if (!res.body) {
      throw new Error('No response body');
    }
    let result = '';
    const reader = res.body.getReader();

    while (true) {
      const { done, value } = await reader.read();
      if (done) {
        break;
      }
      result += decodeAIStreamChunk(value);
    }

    setResponse(result);
  };


  return (
    <div className='mx-auto w-full max-w-md py-24 flex flex-col stretch'>
      <form onSubmit={(e) => handleSubmit(e)}>
        <input
          className='w-full max-w-md bottom-0 border border-gray-300 rounded mb-8 shadow-xl p-2'
          value={input}
          placeholder='Say something...'
          onChange={(e) => setInput(e.target.value)}
          style={{ color: 'black' }}
        />
      </form>

      {<div>
        {response}
      </div>}
    </div>
  );
}

The route handler code I tested is also very similar to the example given in the Vercel AI SDK docs. It looks something like this:

...

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY
});
const openai = new OpenAIApi(config);

export const runtime = 'edge';

export async function POST(req: Request) {
  const { messages, functions } = await req.json();

  // Note that gpt-3.5-turbo-0613 MUST be used if functions are being passed in the request.
  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo-0613',
    stream: true,
    messages,
    functions
  });

  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream);
}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 028a653

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[stephenkalnoske]

@jaredpalmer anyone at Vercel have eyes on this? Right now there's no way to stream in function responses from OpenAI. Pretty bad timing for sure that you released just as OpenAI released new models and functionality, sorry about that.

[CarlosZiegler]

Please take a look on the answer, I think the Vercel are working to solve it:
#140

[stephenkalnoske-sans]

Please take a look on the answer, I think the Vercel are working to solve it: #140

I missed that, good to hear. Thanks!

[Zakinator123]

Closing this PR in favor of #178