Is using `experimental_attachments` required? `file` `parts` aren't working.

[pieterbeulque]

Description

I'm doing a pretty standard streamText on the back-end with useChat() on the front-end.

If I send this from the client, the LLM tells me it can't access any files.

append({
  role: "user",
  content: "what is in this file?"
  parts: [
    { type: "text", text: "what is in this file?" },
    { type: "file", mimeType: "application/pdf", data: "data:application/pdf;base64,…" }
  ]
})

but if I do this

append({
  role: "user",
  content: "what is in this file?"
  parts: [
    { type: "text", text: "what is in this file?" }
  ]
},
{
  experimental_attachments: [
    { name: "file", contentType: "application/pdf", data: "data:application/pdf;base64,…" }
  ]
})

it works.

I'm probably doing something wrong, but shouldn't the former also work?

AI SDK Version

ai@4.3.16
@ai-sdk/google@1.2.19

[pieterbeulque]

I think this is a logical consequence of these lines:

ai/packages/ai/core/prompt/convert-to-core-messages.ts

Lines 55 to 66 in 0235765

	const textParts = message.parts
	.filter(part => part.type === 'text')
	.map(part => ({
	type: 'text' as const,
	text: part.text,
	}));
	coreMessages.push({
	role: 'user',
	content: experimental_attachments
	? [...textParts, ...attachmentsToParts(experimental_attachments)]
	: textParts,

Which feels counterintuitive to the types of CoreMessage that still exist?

If I run generateText with these parts, that all still works, so I'm confused as to why we can't pass these from useChat (anymore?).

[pieterbeulque]

If I update my chat endpoint with this to append the filtered-out file parts, it works perfectly fine.

    messages: convertToCoreMessages(messages).map((m, i) => {
      const originalMessage = messages[i];

      if (m.role !== "user") {
        return m;
      }

      if (!Array.isArray(m.content)) {
        return m;
      }

      const originalParts = originalMessage?.parts ?? [];

      return {
        ...m,
        content: [
          ...m.content,
          ...originalParts.filter((p) => p.type === "file"),
        ],
      };
    }),

[pieterbeulque]

I can see a User messages can have text parts comment on UIMessage but it feels an unnecessary constraint to me, if the file type parts are how it works in CoreMessage.

[lgrammel]

AI SDK 5 replaces the attachments mechanism with file parts on user messages. We are unlikely to make changes to the v4 attachments mechanism since it was experimental.

[pieterbeulque]

@lgrammel An experimental feature shouldn't affect users that don't use the experimental feature, which is not the case here (if you use type: "file" parts you're still affected).

Anyway, all good, appreciate the hard work, and another reason to upgrade to v5, I guess 😁