ai/core: support full array element streaming in streamObject

[bernaferrari]

Feature Description

I have a very very very specific use case. I ask for alt images, and when they are done, I retrieve data from unsplash and cache them. The issue is, with the great auto-closing JSON behavior from streamObject, I don't know if alt has completed (received closing ") or not. I wish I had access to the raw (unparsed) text response from GPT, before Vercel AI SDK auto-complete and types it, so I can check if alt has finished processing or not.

Proposal: would be nice to the behaviour where only "finished strings are appended to the object" feels like a good solid alternative too. I think people would love to have the choice between criticalness and speed. Could even be a gzip 1-9 thing: "letter" | "word" | "object". Default is "letter", I want "word" (property), some people might prefer object (after current object has finished). I think it makes sense to implement these variations.

[bernaferrari]

Is there a way to know the stream is done?

This is my script for parsing everything except the last children, so it streams but avoids the bug I described. It is not optimized, but does the job. I welcome improvements, so I don't need to call statusStream.update() outside of the loop:

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children: z.array(z.object({})),
  }),
  system: `You should do this. Follow the API schema here: ...`,
});

let fullJson;
let partialJson = {};
for await (const partialObject of stream.partialObjectStream) {
  fullJson = partialObject;

  const pJson = modifyJsonByRemovingLastNode(fullJson as any);

  if (JSON.stringify(pJson) === JSON.stringify(partialJson)) {
    continue;
  }
  partialJson = pJson as any;

  const debug = await prettier.format(JSON.stringify(fullJson), {
    parser: "json-stringify",
  });
  console.log('debug info', debug);

 const content = await executeMethod(partialJson);

  statusStream.update({
    type: "loading",
    content,
  });
}

const content = await executeMethod(fullJson);
statusStream.update({
  type: "loading",
  data: content
});

statusStream.done();

[lgrammel]

A possible solution could be to introduce a "fixMode" property on streamObject (or something like it) with the following settings:

• always: always try to fix the JSON. Will lead to incomplete strings etc
• finished-primitive: only fix when primitive attribute values are finished
• finished-objects: only fix when objects are complete. this might need refinement, e.g. for array streaming

I want to think more about all possible cases before adding this, since it can become more complex.

Is my understanding of your use case correct?

Use case:

You want to stream an array of objects that contains an alt attribute (or event an array of strings). 
For each finish array object, partialObjectStream should contain a new result, 
but not for any intermediates (that may have partial urls etc).

[bernaferrari]

Yes. Right now I went lazy and I am with the "I just want to stream the objects that are complete" so there are no surprises anywhere. Like, it might do style="out" instead of style="outlined". Waiting for the whole object to complete is fine for me, while still streaming. The algorithm I did above could be optimized here and there, but it is what I'm doing.

[lgrammel]

First improvement: promise with final, typed object: #1858

[Marviel]

Another idea -- nested newItemCallback object, allowing for subscriptions to items at varying depths

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children: z.array(z.object({})),
  }),
  system: `You should do this. Follow the API schema here: ...`,
  newItemCallbacks: {
    // This function is only called once an item (an object key, or an array item) is complete on the corresponding key path in the schema.
    children: (newItem) => {
      console.log(newItem)
    }
  }
});

Another example:

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children1: z.array(z.object({})),
    nested: z.object({
      children2: z.array(z.object({}))
    })
  }),
  system: `You should do this. Follow the API schema here: ...`,
  newItemCallbacks: {
    // This function is only called once an item (an object key, or an array item) is complete on the corresponding key path in the schema.
    children: (newItem) => {
      console.log(newItem)
    },
    nested: {
      children2: (newChildren2) => {
        console.log(newChildren2)
      }
    }
  }
});

[bernaferrari]

"array" and elementStream is very close to what I imagined.

[matijagrcic]

output-strategy-array

twittervid.com_nicoalbanese10_5efc17.mp4

Ref: https://x.com/nicoalbanese10/status/1831383597304910044

[bernaferrari]

The only possible issues with this would be nested object inside the array, or wanting to have additional fields (like reason) before the array. I hope the upcoming intermediate feature will help with that.

[lgrammel]

@bernaferrari the reasoning before the array is an important limitation. would you mind opening a new ticket about that issue?