Tool result able to return an array of ContentBlock

[tryvin]

So, multimodal LLMs support a tool called results with an array of contents, so it's either content as a string, or content as an array of {type: ....}.

My idea is to add support for the tools to return an array of ContentBlocks, maybe return a Message object?! (ignoring id, role, etc), And then at the provider level, it could implement it, to make sure we are backwards compatible, I thought of introducing a new method getResultAsMessage() that would do if ( result instanceof Message ) return result else new Message(type: text, result...), and rewrite the getResult to extract the text parts of the getResultAsMessage().

That way, providers that rely on getResult continue working (with the drawback of not "properly supporting the message mapping"), and the project providers can expose the actual content to the LLM.

[tryvin]

@ilvalerione Ideas? I held off on creating a PR until you could review the backwards-compatible approach.

[ilvalerione]

Do you have a link to the documentation for multimodal providers asking for this pattern?

[tryvin]

@ilvalerione I told I would hold off creating the PR but went ahead anyway 🙃.

https://developers.openai.com/api/reference/resources/chat#(resource)%20chat.completions%20%3E%20(model)%20chat_completion_content_part_text%20%3E%20(schema)

https://ai.google.dev/api/caching?hl=pt-br#FunctionResponsePart