-
Notifications
You must be signed in to change notification settings - Fork 565
Description
Describe the bug
When sending a message containing a base64 image (e.g., data:image/jpeg;base64,...) using NewMediaPart, Genkit duplicates the content in the OpenAI ChatCompletion request:
- Once as a normal user message (
UserMessage) - Again as an
ImageContentPart
This happens because the same media part is added both to the text content and the image content part.
To Reproduce
Steps to reproduce the behavior. Code samples.
https://genkit.dev/docs/models/?lang=go#multimodal-input-1
image, err := os.ReadFile("photo.jpg")
if err != nil {
log.Fatal(err)
}
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("openai/gpt-4.1-nano"),
ai.WithMessages(
ai.NewUserMessage(
ai.NewMediaPart("image/jpeg", "data:image/jpeg;base64,"+base64.StdEncoding.EncodeToString(image)),
ai.NewTextPart("Compose a poem about this image."),
),
),
)Expected behavior
Base64 image content should appear only once in the ChatCompletion request, as an ImageContentPart. It should not be duplicated as normal user text.
Screenshots
If applicable, add screenshots to help explain your problem.
base64 content in role: user and image_url is same content
Runtime (please complete the following information):
- OS: [e.g. Linux, MacOS]
- Version [e.g. 22]
** Go version
- run
go versionat paste here
Additional context
Add any other context about the problem here.
https://github.com/firebase/genkit/blob/main/go/plugins/compat_oai/generate.go#L116-L135
oaiMessages = append(oaiMessages, openai.UserMessage(content))
parts := []openai.ChatCompletionContentPartUnionParam{}
for _, p := range msg.Content {
if p.IsMedia() {
part := openai.ImageContentPart(
openai.ChatCompletionContentPartImageImageURLParam{
URL: p.Text,
})
parts = append(parts, part)
continue
}
}
if len(parts) > 0 {
oaiMessages = append(oaiMessages, openai.ChatCompletionMessageParamUnion{
OfUser: &openai.ChatCompletionUserMessageParam{
Content: openai.ChatCompletionUserMessageParamContentUnion{OfArrayOfContentParts: parts},
},
})
}
Metadata
Metadata
Assignees
Labels
Type
Projects
Status