[Platform][Azure OpenAI] Image message content serialization incorrect due to Normalizer ordering #488
Description
Summary
When I try to send an image to Azure OpenAI endpoint, I get an error of Response does not contain choices..
This is a red herring to the actual problem, which is an error response from the API:
"error": {
"message": "Invalid file data: 'messages[0].content[1].file.file_data'. Expected a base64-encoded data URL with an application/pdf MIME type (e.g. 'data:application/pdf;base64,SGVsbG8sIFdvcmxkIQ=='), but got unsupported MIME type 'image/png'.",
"param": "messages[0].content[1].file.file_data",
}Even though I am attaching an Image, it gets serialized as a plain File. This is because the OpenAiContract prepends its own FileNormalizer, which also matches instances of Image (because subclass and is_a() check).
A quick fix is to explicitly prepend the default ImageNormalizer in front. With that change the reproducer below starts working again. There are no other File subtypes that would be affected (AudioNormalizer is already prepended there).
Reproducer
#[AsCommand(
'debug:ai:chat-image',
)]
class ImageTestChatCommand
{
public function __construct(
#[Autowire(service: 'ai.agent.tester')] private readonly AgentInterface $agent,
) {
}
public function __invoke(InputInterface $input, OutputInterface $output) : int
{
$io = new SymfonyStyle($input, $output);
$messages = new MessageBag();
$messages->add(Message::ofUser(
'What is in the image?',
Image::fromFile('/Users/melka/cat.png'),
));
$response = $this->agent->call(
$messages,
);
$io->success($response->getContent());
return Command::SUCCESS;
}
}