Add image and audio prompting API

[domenic]

Closes #40. Somewhat helps with #70.

[michaelwasserman]

Some models may only accept a single image or single audio input with each request. Consider describing that edge-case detail behavior (e.g. throw a "NotSupportedError" DOMException) when an unsupported number or combination of image/audio prompt pieces are passed. Maybe also give a single-image example here for wider compatibility.

[domenic]

I think we should encapsulate that away from the user so they don't have to worry about it, by sending two requests to the backend.

[sushraja-msft]

I think number of images aspect of this can be handled through the contextoverflow event. https://github.com/webmachinelearning/prompt-api#tokenization-context-window-length-limits-and-overflow.

Phi 3.5 vision https://huggingface.co/microsoft/Phi-3.5-vision-instruct, recommends at the most 16 images but passing more (say 17) will only run into context length limitations. The context length limitation can be reached earlier as well with large image.

Sending two requests to the backend may not work though, the assistant is going to add its response in between.

Are there other limitations in terms of no mixing of images/audio or limitations in number of images/audio. Ill have to check what the server-side models do for their API.

[domenic]

Phi 3.5 vision https://huggingface.co/microsoft/Phi-3.5-vision-instruct, recommends at the most 16 images but passing more (say 17) will only run into context length limitations.

I agree with you that contextoverflow is the right way to handle this. This seems very natural to me. Each image takes up some of the token budget.

I'm hesitant to add additional "artificial" restrictions on top of the existing context limit restriction. Hopefully that won't be necessary.

Sending two requests to the backend may not work though, the assistant is going to add its response in between.

At least for some architectures, this should be avoidable, by not sending the control token that triggers a response from the model.

Are there other limitations in terms of no mixing of images/audio or limitations in number of images/audio. Ill have to check what the server-side models do for their API.

I've put Deep Research on this question, and we should hear back soon...

[domenic]

Deep Research report. Some takeaways:

• Generally multiple images are fine, and mixing modalities is fine. (Although the report seemed to imply Llama might be more single-turn based.)
• There seems to be some hallucination about only one audio per request. Clicking through the provided sources I cannot find backing for this. So, probably audio is fine too, but we might need to keep an eye out.
• File size limits are pretty generous, but they do exist.
• Pixel size limits are O(1k x 1k), but rescaling is the common solution, sometimes provided automatically. For the web I think automatic rescaling is a good idea.

[beaufortfrancois]

Shall we use HTMLMediaElement instead and fail if there's no audio track enabled?

[domenic]

What would be the benefit of doing that?

[beaufortfrancois]

Web developers may want to analyse audio data from a video file. Adding HTMLMediaElement support would allow them to do that without having to use a specific HTMLAudioElement for this.

My question would be more why disallowing audio inputs from HTMLVideoElement in the first place when it actually support audio?

[domenic]

I think it might be confusing to take video as input and only pull out the audio component. Do you know of any other web platform APIs that do that?

[beaufortfrancois]

WebAudio AudioContext createMediaElementSource() and HTMLMediaelement setsinkid() are two examples.
To a certain extent, HTMLMediaelement capturestream() also applies.

[domenic]

Hmm, those don't seem quite the same to me, because the video content isn't completely discarded, right? In those cases, the audio is re-routed, but the video keeps playing.

[beaufortfrancois]

I got confused. Thanks for explaining, I now see where those examples don't match what we're trying to achieve with this proposal for HTMLAudioElement support.
The audio data in HTMLAudioElement will be downloaded completely (if necessary) while both video and audio data from an HTMLMediaElement should ideally not be downloaded completely, only the audio component.

Out of curiosity, is there a web platform API that also pull out the audio component of HTMLAudioElement, or for that matter of an HTMLMediaElement?

[domenic]

You raise a good point that this is unprecedented, and so I'm now a bit less sure that including HTMLAudioElement is a good idea.

To confirm, I went through all references to HTMLMediaElement and to HTMLAudioElement in the spec ecosystem and it looks like none of them consume the element in such a way.

In fact, doing so made me more hesitant about this because so many specs deal with patching HTMLMediaElement to make it "more streaming", e.g. streaming it to a web audio context, or to or from a realtime A/V stream.

The one spec that is all about consuming such data, the MediaRecorder spec, actually takes a MediaStream.

The only argument for accepting HTMLAudioElement I have is that to me, HTMLAudioElement is in between HTMLImageElement (which we accept per the precedent set by createImageBitmap()) and HTMLVideoElement. In all three cases, streaming is used to download them. In the image case, streaming to completion happens automatically once the image starts loading. For the video case, that's generally a bad idea, and we need strong incremental streaming support. Audio feels somewhere in between, where by default it streams incrementally, but streaming to completion isn't "as bad", at least on modern internet. So since we want to allow accepting a HTMLImageElement for image prompts, maybe we should accept a HTMLAudioElement for audio prompts?

But on balance perhaps it'd be best to leave out HTMLAudioElement for now, until we hear from developers that it'd be helpful for their use cases. My (not very informed) guess is that most audio sources will end up being Blobs or similar from fetch() anyway. WDYT?

[beaufortfrancois]

I agree with you that removing HTMLAudioElement support for now is probably best. We should re-add it when the API actually supports streaming data through MediaStreamTrack, like Web Speech.

As you said, web developers will be able to use fetch() for simple cases:

const audio = document.querySelector('audio');
const audioBlob = await (await fetch('file.mp3').response()).blob();

const response = await session.prompt([
  "My response to your critique:",
  { type: "audio", content: audioBlob }
]);

[bil-ash]

Would be nice if someone would provide a multimodal example for the prompting API because all the examples I have seen till now are text-only.

[bradtriebwasser]

Would be nice if someone would provide a multimodal example for the prompting API because all the examples I have seen till now are text-only.

@bil-ash The README.md outlines some multimodal examples. Is there anything specific you are looking for?