Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ai/core: support full array element streaming in streamObject #1486

Closed
bernaferrari opened this issue May 3, 2024 · 9 comments
Closed

ai/core: support full array element streaming in streamObject #1486

bernaferrari opened this issue May 3, 2024 · 9 comments
Assignees
Labels
ai/core enhancement New feature or request

Comments

@bernaferrari
Copy link

bernaferrari commented May 3, 2024

Feature Description

I have a very very very specific use case. I ask for alt images, and when they are done, I retrieve data from unsplash and cache them. The issue is, with the great auto-closing JSON behavior from streamObject, I don't know if alt has completed (received closing ") or not. I wish I had access to the raw (unparsed) text response from GPT, before Vercel AI SDK auto-complete and types it, so I can check if alt has finished processing or not.

Proposal: would be nice to the behaviour where only "finished strings are appended to the object" feels like a good solid alternative too. I think people would love to have the choice between criticalness and speed. Could even be a gzip 1-9 thing: "letter" | "word" | "object". Default is "letter", I want "word" (property), some people might prefer object (after current object has finished). I think it makes sense to implement these variations.

@bernaferrari bernaferrari changed the title I miss rawText in streamObject I miss rawText in streamObject May 3, 2024
@bernaferrari
Copy link
Author

bernaferrari commented May 3, 2024

Is there a way to know the stream is done?

This is my script for parsing everything except the last children, so it streams but avoids the bug I described. It is not optimized, but does the job. I welcome improvements, so I don't need to call statusStream.update() outside of the loop:

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children: z.array(z.object({})),
  }),
  system: `You should do this. Follow the API schema here: ...`,
});

let fullJson;
let partialJson = {};
for await (const partialObject of stream.partialObjectStream) {
  fullJson = partialObject;

  const pJson = modifyJsonByRemovingLastNode(fullJson as any);

  if (JSON.stringify(pJson) === JSON.stringify(partialJson)) {
    continue;
  }
  partialJson = pJson as any;

  const debug = await prettier.format(JSON.stringify(fullJson), {
    parser: "json-stringify",
  });
  console.log('debug info', debug);

 const content = await executeMethod(partialJson);

  statusStream.update({
    type: "loading",
    content,
  });
}

const content = await executeMethod(fullJson);
statusStream.update({
  type: "loading",
  data: content
});

statusStream.done();

@lgrammel
Copy link
Collaborator

lgrammel commented May 7, 2024

A possible solution could be to introduce a "fixMode" property on streamObject (or something like it) with the following settings:

  • always: always try to fix the JSON. Will lead to incomplete strings etc
  • finished-primitive: only fix when primitive attribute values are finished
  • finished-objects: only fix when objects are complete. this might need refinement, e.g. for array streaming

I want to think more about all possible cases before adding this, since it can become more complex.

Is my understanding of your use case correct?

Use case:

You want to stream an array of objects that contains an alt attribute (or event an array of strings). 
For each finish array object, partialObjectStream should contain a new result, 
but not for any intermediates (that may have partial urls etc).

@lgrammel lgrammel changed the title I miss rawText in streamObject ai/core: support full array element streaming in streamObject May 7, 2024
@lgrammel lgrammel self-assigned this May 7, 2024
@bernaferrari
Copy link
Author

bernaferrari commented May 7, 2024

Yes. Right now I went lazy and I am with the "I just want to stream the objects that are complete" so there are no surprises anywhere. Like, it might do style="out" instead of style="outlined". Waiting for the whole object to complete is fine for me, while still streaming. The algorithm I did above could be optimized here and there, but it is what I'm doing.

@lgrammel
Copy link
Collaborator

lgrammel commented Jun 6, 2024

First improvement: promise with final, typed object: #1858

@lgrammel lgrammel added the enhancement New feature or request label Jun 6, 2024
@Marviel
Copy link

Marviel commented Aug 15, 2024

Another idea -- nested newItemCallback object, allowing for subscriptions to items at varying depths

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children: z.array(z.object({})),
  }),
  system: `You should do this. Follow the API schema here: ...`,
  newItemCallbacks: {
    // This function is only called once an item (an object key, or an array item) is complete on the corresponding key path in the schema.
    children: (newItem) => {
      console.log(newItem)
    }
  }
});

Another example:

const stream = await streamObject({
  model: openai.chat("gpt-4-turbo"),
  schema: z.object({
    children1: z.array(z.object({})),
    nested: z.object({
      children2: z.array(z.object({}))
    })
  }),
  system: `You should do this. Follow the API schema here: ...`,
  newItemCallbacks: {
    // This function is only called once an item (an object key, or an array item) is complete on the corresponding key path in the schema.
    children: (newItem) => {
      console.log(newItem)
    },
    nested: {
      children2: (newChildren2) => {
        console.log(newChildren2)
      }
    }
  }
});

@bernaferrari
Copy link
Author

"array" and elementStream is very close to what I imagined.

@lgrammel lgrammel closed this as completed Sep 5, 2024
@matijagrcic
Copy link

output-strategy-array

twittervid.com_nicoalbanese10_5efc17.mp4

Ref: https://x.com/nicoalbanese10/status/1831383597304910044

@bernaferrari
Copy link
Author

The only possible issues with this would be nested object inside the array, or wanting to have additional fields (like reason) before the array. I hope the upcoming intermediate feature will help with that.

@lgrammel
Copy link
Collaborator

lgrammel commented Sep 9, 2024

@bernaferrari the reasoning before the array is an important limitation. would you mind opening a new ticket about that issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ai/core enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants