Images for Vision in the Assistants API

[andehr]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to look for a similar issue and didn't find it.
• I searched the Marvin documentation for this feature.

Describe the current behavior

Currently when building a Thread for use with an Assistant assistant, you can add messages with the .add() function, which takes text content, role, code_interpreter_files, and file_search_files. But as of May 9th, Open AI's Assistant API supports adding images to threads. The images should be uploaded with the purpose "vision" (rather than "assistants" used for code_interpreter_files and file_search_files).

Describe the proposed behavior

Perhaps an additional parameter image_files which accepts image files, uploads them, then attaches their file IDs to the thread.

Though it looks like the images may be added as content parts rather than attachments?

Perhaps there should be an option to directly pass known file IDs instead of uploading the file every time?

Example Use

# new thread
thread = Thread()

# add message with image attachment
with open("path/to/image.png", "rb") as f:
    thread.add("Ask me questions about this image", image_files=[f])

# run thread
thread.run(assistant=some_assistant)

Additional context

In general I've found that the latest open ai models perform very well with threads of images and text. Being able to deal with multiple image, texts, file search together with all the benefits of thread management, would be super helpful for my use cases.

[andehr]

Maybe a change like this to the Thread.add_async() would be the minimal?

    @expose_sync_method("add")
    async def add_async(
        self,
        message: str,
        role: str = "user",
        code_interpreter_files: Optional[list[str]] = None,
        file_search_files: Optional[list[str]] = None,
        image_files: Optional[list[str]] = None,
    ) -> Message:
        """
        Add a user message to the thread.
        """
        client = marvin.utilities.openai.get_openai_client()

        if self.id is None:
            await self.create_async()

        content = [dict(text=message, type="text")]

        # Upload files and collect their IDs
        attachments = []
        for fp in code_interpreter_files or []:
            with open(fp, mode="rb") as file:
                response = await client.files.create(file=file, purpose="assistants")
                attachments.append(
                    dict(file_id=response.id, tools=[dict(type="code_interpreter")])
                )
        for fp in file_search_files or []:
            with open(fp, mode="rb") as file:
                response = await client.files.create(file=file, purpose="assistants")
                attachments.append(
                    dict(file_id=response.id, tools=[dict(type="file_search")])
                )
        for fp in image_files or []:
            with open(fp, mode="rb") as file:
                response = await client.files.create(file=file, purpose="vision")
                content.append(
                    dict(image_file=dict(file_id=response.id), type="image_file")
                )

        # Create the message with the attached files
        response = await client.beta.threads.messages.create(
            thread_id=self.id, role=role, content=content, attachments=attachments
        )
        return response

[zzstoatzz]

thanks for the great issue @andehr! looking at the PR now