[Bug]: Using the dall-e 3 notebook example code, the request data is too large.

[deerleo]

Describe the bug

I tried to integrate the dalle-3 example code to my groupchat bot, (https://github.com/microsoft/autogen/blob/main/notebook/agentchat_dalle_and_gpt4v.ipynb),
but after the dalle agent replied with img data in base64 format, the next request encounter an error sad request data too large.
I think it's because the autogen push the reply data of base64 img to message list, in the next request, there will be a large request body.

How to exclude some message or truncate it in the group chat?

Steps to reproduce

No response

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

[rickyloynd-microsoft]

@kevin666aa

[yiranwu0]

Hello @deerleo, when you are creating dalle agent, are you using your own customized agents? A quick solution is to process the image in the customized registered reply function. You will take out the image and return a message like: "Placeholder for Image".

[sonichi]

@BeibinLi

[BeibinLi]

@deerleo Thanks for the feedback. I tried the notebook again and it works ok.

Can you provide a simple example to reproduce the error? My guess is the Chat Manager is not a LMM agent (e.g., it uses a LLM with no vision features). So, it can not understand the b64 image format, and read the image as a super long string.

I will try to redesign how LMM agents handle images in the future. In the meanwhile, if you could provide some failing examples, that would be great!

[BeibinLi]

@deerleo Can you check: #1124

Also, can you make the chat_manager a MultimodalAgent instead of a conversable agent?

[afourney]

@BeibinLi I don't think there is a MultimodalGroupChat?

You can add groupchat manager capabilities to any agent via registration, but I suspect the problem lies with the GroupChat class that handles the orchestration.

[BeibinLi]

@afourney Got it. You are correct. I just added an issue (#1142) regarding MultimodalGroupChat. I will create a MultimodalGroupChat under the multimodal features.