[Feature]: `gemini-embedding-2-preview` multimodal input support (image/video/audio/PDF) in `batchEmbedContents`

[ruettiger]

Feature Description

gemini-embedding-2-preview is Google's first natively multimodal embedding model, supporting text, images, video, audio, and PDF inputs. However, the current LiteLLM implementation only handles text-only input — all other modalities are silently treated as text strings.

Current Behavior

In litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_transformation.py, the transform_openai_input_gemini_content function wraps all inputs as PartType(text=i):

def transform_openai_input_gemini_content(input, model, optional_params):
    for i in input:
        request = EmbedContentRequest(
            model=gemini_model_name,
            content=ContentType(parts=[PartType(text=i)]),  # ← always text
            **optional_params
        )

This means:

• Base64-encoded images → sent as text string (garbage embedding)
• data:image/png;base64,... URIs → sent as text string
• Video/audio references → sent as text string
• Mixed text+image inputs → not supported

Expected Behavior

The function should detect input type and construct appropriate PartType:

Input format	Should produce
Plain string	PartType(text=input)
data:image/png;base64,...	PartType(inline_data=BlobType(mime_type="image/png", data=base64_data))
Raw base64 (detected)	PartType(inline_data=BlobType(mime_type="image/png", data=base64_data))
gs://... GCS URI	PartType(file_data=FileDataType(mime_type=..., file_uri=uri))
Dict {"text": ..., "inline_data": ...}	Multi-part content with both text and media

Type Definitions Already Support This

The PartType TypedDict already has the necessary fields:

class PartType(TypedDict, total=False):
    text: str
    inline_data: BlobType      # ← for base64 images/audio
    file_data: FileDataType    # ← for GCS URIs

And BlobType:

class BlobType(TypedDict, total=False):
    mime_type: Required[str]
    data: Required[str]  # base64-encoded

Supported Modalities per Google Docs

Per Google's documentation:

• Text: up to 8,192 tokens
• Images: up to 6 per request (PNG, JPEG)
• Video: max 2 minutes (MP4, MOV)
• Audio: max 80 seconds (MP3, WAV)
• PDF: up to 6 pages per file

Motivation

Without this, gemini-embedding-2-preview in LiteLLM is essentially a text-only embedding model, missing its primary differentiator — native multimodal support. Users who want image/video/audio embeddings have to bypass LiteLLM entirely.

Suggested Implementation

1. Add input-type detection logic in transform_openai_input_gemini_content (similar to what VertexAIMultimodalEmbeddingConfig._process_input_element does for the old multimodalembedding model)
2. Construct PartType with appropriate fields based on detected input type
3. Support data:mime_type;base64,... URI format for inline media
4. Support dict inputs like {"inline_data": {"mime_type": "image/png", "data": "base64..."}} for explicit specification

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

main branch

Twitter / LinkedIn details

No response

[chiruno-9]

That explains why our tests on this model turn out to have terribly bad results. 💀

[UnlastingR]

Is there any progress here?

[GGJJack]

I'm having the same issue.

[chiruno-9]

Confirmed working on LiteLLM proxy v1.82.3 (patch 4)

We tested vertex_ai/gemini-embedding-2-preview extensively on proxy v1.82.3p4 and can confirm:

Text embedding — works. The :predict endpoint routing issue is resolved; the model now correctly
goes through GoogleBatchEmbeddings.

Image embedding via base64 data URI — works. Passing data:image/png;base64,... in the input
field correctly sends inline_data to Gemini (not as a text string). We verified by comparing
proxy output against the google-genai SDK directly — embeddings are identical to 4 decimal
places.

Interleaved text+image — works. Passing input=["text", "data:image/...;base64,..."] produces a
single fused embedding that blends both modalities, matching the genai SDK behavior.

One minor issue: passing multiple strings in input does not batch-embed them independently — all
items get fused into a single Content with multiple parts, returning 1 embedding instead of N.
This means you need to call once per item for independent embeddings. Not a blocker, but worth
noting for batch use cases.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.