Uploading PDF documents as part of the Prompt to OpenAI

[ychaikin]

Expected Behavior
Perhaps I am not doing this correctly. If so, please point me in the direction.

OpenAI GPT-4 and on allow you to upload a PDF as part of the chat request. So, something like this should be possible:

byte[] fileData = ...
Media media = Media.builder().data(fileData).mimeType(DOC_PDF).build();
Prompt prompt = new Prompt(
        List.of(new UserMessage("extract text from the following PDF document."),
            new UserMessage("some text here", media)), openAiChatOptions);

CallResponseSpec callResponseSpec = chatClient.prompt(prompt).call();
ChatResponse chatResponse = callResponseSpec.chatResponse();

Current Behavior

While the above compiles, even with GPT-4o (which supports this), I get the following error:

org.springframework.ai.retry.NonTransientAiException: 400 - {
  "error": {
    "message": "Invalid MIME type. Only image types are supported.",
    "type": "invalid_request_error",
    "param": null,
    "code": "invalid_image_format"
  }
}

Pretty clear - can't use a PDF document as part of a UserMessage. I am not currently aware of a workaround, but if there is one, please point me to it.

Much appreciated!

[dev-jonghoonpark]

OpenAI GPT-4 and on allow you to upload a PDF as part of the chat request

Where did you find this information?

Based on the API specification, it seems that only text, image, and audio inputs are allowed.

[dev-jonghoonpark]

I've been looking for information on this and it looks like it's possible using the assistants api.

• related doc
  • https://platform.openai.com/docs/api-reference/assistants
  • https://stackoverflow.com/a/78924474/29672082

Currently, the assistants api is in beta.
As far as I know, spring ai does not support the assistants api yet.

[ychaikin]

Here is the page that shows you a couple of ways of doing, including the one I was interested in, which is embedding base64 encoded...
https://platform.openai.com/docs/guides/pdf-files?api-mode=responses

Under Supported Models it says this:
Only models that support both text and image inputs, such as gpt-4o, gpt-4o-mini, or o1, can accept PDF files as input.

[dev-jonghoonpark]

Thank you for the information.
I think we'd probably need to add a feature to make it work that way

[dev-jonghoonpark]

related issue : #2962

[markpollack]

BTW, the assistants api is going to be deprecated. As per the openai page

Based on your feedback from the Assistants API beta, we've incorporated key improvements 
into the Responses API. After we achieve full feature parity, we will announce a deprecation
 plan later this year, with a target sunset date in the first half of 2026. 
[Learn more](https://platform.openai.com/docs/guides/responses-vs-chat-completions).

[markpollack]

There are some docs on this - https://docs.spring.io/spring-ai/reference/api/multimodality.html#_spring_ai_multimodality

also test case here -

spring-ai/models/spring-ai-openai/src/test/java/org/springframework/ai/openai/chat/client/OpenAiChatClientIT.java

Line 308 in bfcaad7

void multiModalityEmbeddedImage(String modelName) throws IOException {

if that doesn't solve your problem, please reopen the issue.

[ychaikin]

@markpollack IN the IT that you referenced, this line:
.media(MimeTypeUtils.IMAGE_PNG, new ClassPathResource("/test.png")

The MimeTypeUtils class doesn't have PDF as an option. Trying to substitute this:

.media(MimeType.valueOf("application/pdf"),
                new ClassPathResource("/file.pdf")

(MypeType is org.springframework.util.MimeType)

When I do that, I get the same error response back:

org.springframework.ai.retry.NonTransientAiException: 400 - {
  "error": {
    "message": "Invalid MIME type. Only image types are supported.",
    "type": "invalid_request_error",
    "param": null,
    "code": "invalid_image_format"
  }
}

Also, I do not have permissions to reopen the issue.