Use streamText with tools causes TypeValidationError errors

[ethanmick]

Hey everyone, I'm trying to use ai without using the frontend code, useChat. The examples that follows are Node.js/backend only. Also, I'm using an openai compatible API, but the model is not an OpenAI model. Not sure if that matters or not. And "compatible" might be the problem.

What I want to do: Stream a full response that includes tool calls and have the text + tool calls correctly handled. When I execute the sample code below, the text stream is correctly handled as expected, but then when the LLM goes to call my function, the arguments themselves are streamed to the client. Each argument represents a few bytes and so each one fails the schema validation.

Output:

Type step-start
I'll get the current weather in Boston for you in Fahrenheit.

Validation Error Delta { name: 'getWeather', arguments: '' }
Validation Error Delta { arguments: '' }
Validation Error Delta { arguments: '{"city": "' }
Validation Error Delta { arguments: 'Boston"' }
Validation Error Delta { arguments: ', "un' }
Validation Error Delta { arguments: 'it": "' }
Validation Error Delta { arguments: 'F"}' }

Type step-finish
Type finish
Calls []
Results []

The LLM is trying to call the getWeather function, and providing the correct arguments, but because each chunk only has a partial chunk of the entire schema, they all fail the schema validation. I've tried various options in the streamText but can't get it work as expected.

I'm looking for the obvious thing I'm missing - I feel like streaming the response and expecting tool calls within it is table stakes for tool functionality, but my minimal example isn't working.

This does work as expected if I use generateText instead.

Sample Code

async function main() {
  const messages: Message[] = [
    {
      role: 'user',
      content: 'What is the weather in Boston in F?',
    },
  ]

  const result = streamText({
    model: openai(''), // custom model setup here, openai compatible AI.
    system: 'You are a helpful assistant.',
    messages,
    maxSteps: 5,
    toolCallStreaming: true,
    tools: {
      getWeather: {
        description: 'Get the weather for a location',
        parameters: z.object({
          city: z.string().describe('The city to get the weather for'),
          unit: z
            .enum(['C', 'F'])
            .describe('The unit to display the temperature in'),
        }),
        execute: async ({ city, unit }) => {
          const weather = {
            value: 24,
            description: 'Sunny',
          }

          return `It is currently ${weather.value}°${unit} and ${weather.description} in ${city}!`
        },
      },
    },
  })

  for await (const chunk of result.fullStream) {
    if (chunk.type === 'text-delta') {
      process.stdout.write(chunk.textDelta)
    } else if (chunk.type === 'error') {
      console.log(chunk.error)
      console.log(
        'Validation Error Delta',
        chunk.error?.value?.choices?.[0]?.delta?.tool_calls?.[0]?.function
      )
    } else {
      console.log('Type', chunk.type)
    }
  }
}

main()

Thanks!

[ethanmick]

Alright, running this again openai itself and gpt-4o correctly creates the output:

$ npx tsx stream-tool.ts
Type step-start
Type tool-call-streaming-start
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call-delta
Type tool-call
Type tool-result
Type step-finish
Type step-start
The current weather in Boston is 24°F and sunny.
Type step-finish
Type finish

So I'm guessing it has to do with the API I'm using for my model. I'm going to keep this open and see if I can get any clarity on the API side what might need to be changed.