Python | How to pass multiple images and refer it in user message

[pintuiitbhi]

My use case is something like this:

User provides a json content (data). eg

{
    "input": "Compare http://example.com/a.jpg and http://example.com/b.jpg based on which is best representation of this data ...."
}

Now the ChatCompletionAgent should read this image. But it is not working.

What is the current approach to handle such use case?

           user_message = ChatMessageContent(
                role=AuthorRole.USER,
                content=json.dumps(data, indent=2)
            )

I tried to create a ImageReaderPlugin and added to Kernel but it gets token max limit when sending to llm api.

import base64
from typing import Annotated
from semantic_kernel.functions import kernel_function

class ImageReaderPlugin:
    """Plugin to read images and convert them to base64."""
    
    @kernel_function(
        name="read_image_as_base64",
        description="Reads an image file from the filesystem and converts it to base64 format. Use this tool when you need to analyze, view, or process an image file. Returns a base64 encoded string that can be included in LLM messages for vision analysis."
    )
    def read_image_as_base64(
        self,
        file_path: Annotated[str, "The full path to the image file"]
    ) -> Annotated[str, "Base64 encoded string of the image"]:
        """Read image file and return base64 encoded string."""
        try:
            with open(file_path, "rb") as image_file:
                return base64.b64encode(image_file.read()).decode('utf-8')
        except FileNotFoundError:
            return f"Error: Image file not found at path: {file_path}"
        except Exception as e:
            return f"Error reading image file: {str(e)}"

[pintuiitbhi]

Also what if we have large xlsx or csv files path then what should be the approach.

[moonbox3]

Hi @pintuiitbhi, we have an ImageContent type that can one leverage. You can find an example of using that content type in this sample:

semantic-kernel/python/samples/concepts/chat_completion/simple_chatbot_with_image.py

Line 41 in 8d1b3fd

image_content_remote = ImageContent(uri=IMAGE_URI)

You'd want to be able to format that appropriate content type, and when the messages are formed to send to the model, it will properly handle the information.