AssertionError when sending base64 images to OpenAI Vision Models

[Louisfrn]

Question

Hello pydantic-ai team,

I'm trying to use your library to analyze base64 encoded images with an OpenAI gpt-4o-mini, but I'm consistently running into an AssertionError that originates deep within the library's internal mapping logic.

I have tried formatting the user_prompt as a list of content blocks (text and image), following the standard OpenAI API structure as suggested in issue #126.

In the doc I saw, BinaryContent, should I use it ?

This last approach is what generates the traceback below. It seems the library's internal _map_user_prompt function does not expect to receive a pre-formatted dictionary for the image content and fails when it encounters one.

Could you please clarify the current, recommended method for passing base64 image data to an Agent that uses OpenAIChatModel? Any documentation or examples on this specific use case would be greatly appreciated.

Code to Reproduce

The following code reliably reproduces the error. The analyze_with_base64 method constructs a user_prompt as a list of content blocks, which then causes the agent.run_sync() call to fail.

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
# Other necessary imports...

class ImageAnalyzer:
    def __init__(self, model_name="gpt-4-vision-preview", system_prompt="Analyze the image."):
        self.model = model_name
        self.system_prompt = system_prompt

    def create_agent(self, response_model):
        """Create a Pydantic AI agent with the specified response model."""
        return Agent(
            model=OpenAIChatModel(self.model),
            output_type=response_model,
            system_prompt=self.system_prompt
        )
    
    def analyze_with_base64(self, base64_image, response_model, additional_prompt=""):
        """Analyze base64 encoded image with Pydantic AI."""
        try:
            agent = self.create_agent(response_model)
            image_data_url = f"data:image/jpeg;base64,{base64_image}"
            
            # Build the user prompt as a list of content blocks
            user_prompt_content = [
                {
                    "type": "image_url",
                    "image_url": {"url": image_data_url}
                }
            ]

            if additional_prompt:
                user_prompt_content.insert(0, {
                    "type": "text", 
                    "text": additional_prompt
                })

            # This call triggers the error
            result = agent.run_sync(
                user_prompt=user_prompt_content,
                message_history=[],
                model_settings={
                    'max_tokens': 2000,
                    'temperature': 0
                }
            )
            
            return result.data
            
        except Exception as e:
            # For debugging purposes
            print(f"Error in Pydantic AI analysis: {e}")
            raise

Actual Behavior (Traceback) :

When run_sync is called with the user_prompt formatted as a list, it results in the following error:

AssertionError: Expected code to be unreachable, but got: {'type': 'image_url', 'image_url': {'url': 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDA...'}}

Traceback (most recent call last):
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\datastorm-one-llm\features\analyzers\base\openai_analyzer.py", line 106, in analyze_with_base64
    result = agent.run_sync(
             ^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\agent\abstract.py", line 317, in run_sync
    return get_event_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\agent\abstract.py", line 218, in run
    async for node in agent_run:
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\run.py", line 149, in __anext__
    next_node = await self._graph_run.__anext__()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_graph\graph.py", line 758, in __anext__
    return await self.next(self._next_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_graph\graph.py", line 731, in next
    self._next_node = await node.run(ctx)
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\_agent_graph.py", line 399, in run
    return await self._make_request(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\_agent_graph.py", line 441, in _make_request
    model_response = await ctx.deps.model.request(message_history, model_settings, model_request_parameters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\models\openai.py", line 396, in request
    response = await self._completions_create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\models\openai.py", line 455, in _completions_create
    openai_messages = await self._map_messages(messages)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\models\openai.py", line 646, in _map_messages
    async for item in self._map_user_message(message):
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\models\openai.py", line 726, in _map_user_message
    yield await self._map_user_prompt(part)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\louis\OneDrive - Groupe ESIEA\TECHNOVA\.venv\Lib\site-packages\pydantic_ai\models\openai.py", line 799, in _map_user_prompt
    assert_never(item)
  File "C:\Users\louis\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 2542, in assert_never
    raise AssertionError(f"Expected code to be unreachable, but got: {value}")

Additional Context

No response

[DouweM]

In the doc I saw, BinaryContent, should I use it ?

Yep. It takes bytes, so you can convert your base64 image into bytes using base64.b64decode(...).