[Feature Request] MultimodalPrompt

[chenllliang]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

To enable multimodal perception of agents, we need a flexible multimodal prompt class at the first stage.

The class should have some basic features:

1. flexible to add new modailties.
2. conversion from human-readable to machine-readable format.
3. easy to save and transfer among different agents.

I am willing to add this feature.

Solution

The prompt could have following form:

Listen to this audio {audio} and see the image {image} to describe the scene.

I suggest base64 encoding to store all modality information.

Alternatives

No response

Additional context

No response

[lightaime]

Thanks @chenllliang. That is a great idea! Is there any reference on implementing MultimodalPrompt this way?

[chenllliang]

@lightaime Hi, the design is inspired by current advance in multimodal llm like MMICL and MiniGPT5, which support interleaved text and image as input. The multimodal information could appear on different place of the prompt (at the head of the prompt for most current VLMs). The MultimodalPrompt class can be instantiated according to different multimodal LLM, for example:

1. for MMICL. the input prompt includes the image id and reference information

1. Interleaved Image-Text Data

Input:  Image 0 is <image0> {image 0}
        ...
        Image j is <imagej> {image j}
        {question}

MMICL:  {answer}

2. In−Context Demonstration Data

Input:  Image 0 is <image0> {image 0}.
        {question} 
        {answer} 
        ...
        Image j is <imagej> {image j}.
        {question} 

MMICL:  {answer}

Different VLMs handle the multimodal information differently, MultimodalPrompt should be flexible to fit different models.

[lightaime]

Thanks for the explanation! I think it is promising (Although I have no clue if gpt-4v does it this way). Please feel free to open a pull request. Also happy to discuss more if you want.

[chenllliang]

I don't know either lol, but from the gpt4-v interface provided to user, it seems that gpt4-v simply concats the image and the text.

[lightaime]

I guess they number the images as well. I tried to upload multiple images and asked what is the first one and what is the second one. It does work as expected.

[chenllliang]

thanks for the information