Skip to content

[BUG] - Invalid image data error when using to_solver() with InspectAI evaluations containing inline images #208

@karangattu

Description

@karangattu

Description

When running InspectAI evaluations with datasets containing base64-encoded images (data URLs), the evaluation fails with:

BadRequestError: Error code: 400 - 
{'error': {'message': "The image data you provided does not represent a valid image. Please check your input and try again with one of the supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp'].", 'type': 'invalid_request_error', 'param': 'input', 'code': 'invalid_value'}}

Steps to Reproduce

  1. Create an evaluation dataset with inline images using chat.export_eval()
  2. Use chat.to_solver() to create an InspectAI solver
  3. Run the evaluation with inspect eval

Reproducible code

from chatlas import ChatOpenAI
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_qa

chat = ChatOpenAI(model="gpt-5-nano-2025-08-07")


@task
def my_eval():
    return Task(
        dataset=json_dataset("pic_only_eval_dataset.jsonl"),
        solver=chat.to_solver(),
        scorer=model_graded_qa(),
        model="openai/gpt-5-nano-2025-08-07",
    )

Contents of jsonl file for reference

{"input":[{"id":"TmBUSKu2WyoXGvjFnfEy5F","content":"You can inspect the provided image and document attachments to answer questions accurately. Reference both when forming your response.","role":"system"},{"id":"YyjFMLrekQuCwYnec3fwTU","content":[{"type":"text","text":"Review the attached materials to answer the question."},{"type":"image","image":"https://www.allrecipes.com/thmb/GMjVlmWXRMGPuIK2FRh8MBvIXgA=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/AR-151595-Campbells-green-bean-casserole-DDMFS-4x3-49f408d95f4d40a39cb14ce6fa9544a5.jpg","detail":"auto"},{"type":"text","text":"What is the recipe shown in the image?"}],"role":"user"}],"target":"The recipe shown in the image is a generic Green Bean Casserole."}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions