-
Notifications
You must be signed in to change notification settings - Fork 19
Closed
Description
Description
When running InspectAI evaluations with datasets containing base64-encoded images (data URLs), the evaluation fails with:
BadRequestError: Error code: 400 -
{'error': {'message': "The image data you provided does not represent a valid image. Please check your input and try again with one of the supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp'].", 'type': 'invalid_request_error', 'param': 'input', 'code': 'invalid_value'}}
Steps to Reproduce
- Create an evaluation dataset with inline images using
chat.export_eval() - Use
chat.to_solver()to create an InspectAI solver - Run the evaluation with
inspect eval
Reproducible code
from chatlas import ChatOpenAI
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_qa
chat = ChatOpenAI(model="gpt-5-nano-2025-08-07")
@task
def my_eval():
return Task(
dataset=json_dataset("pic_only_eval_dataset.jsonl"),
solver=chat.to_solver(),
scorer=model_graded_qa(),
model="openai/gpt-5-nano-2025-08-07",
)Contents of jsonl file for reference
{"input":[{"id":"TmBUSKu2WyoXGvjFnfEy5F","content":"You can inspect the provided image and document attachments to answer questions accurately. Reference both when forming your response.","role":"system"},{"id":"YyjFMLrekQuCwYnec3fwTU","content":[{"type":"text","text":"Review the attached materials to answer the question."},{"type":"image","image":"https://www.allrecipes.com/thmb/GMjVlmWXRMGPuIK2FRh8MBvIXgA=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/AR-151595-Campbells-green-bean-casserole-DDMFS-4x3-49f408d95f4d40a39cb14ce6fa9544a5.jpg","detail":"auto"},{"type":"text","text":"What is the recipe shown in the image?"}],"role":"user"}],"target":"The recipe shown in the image is a generic Green Bean Casserole."}Metadata
Metadata
Assignees
Labels
No labels