Skip to content

Bug: Systemic parts[0] indexing drops multimodal streaming data and bypasses content validation #6616

Description

@Harshitmishra001

🔴 Required Information

Describe the Bug:
The framework makes unsafe parts[0] indexing assumptions in the core Gemini Live API connection (gemini_llm_connection.py) and flow modules (base_llm_flow.py). This causes two critical systemic bugs:

  1. Validation Bypass: Mixed-content blocks (e.g., text followed by a tool response) bypass ValueError validation entirely and are incorrectly routed as normal client content.
  2. Dropped Multimodal Streaming: The Live API streaming receiver silently drops subsequent parts in multi-part streaming chunks (e.g., mid-chunk thought boundary transitions or inline data) because it only extracts data from content.parts[0].

Steps to Reproduce:
Please provide a numbered list of steps to reproduce the behavior:

  1. Initialize a GeminiLlmConnection.
  2. Construct a types.Content object with multiple parts, where the first part is text and the second part is a types.FunctionResponse.
  3. Call await gemini_connection.send_content(content).
  4. Observe that the content is routed to the standard send() pipeline instead of correctly raising a ValueError.

Expected Behavior:

  1. Sending a mixed-content block containing a function response should strictly trigger the validation ValueError("Function-response content cannot mix function and non-function parts.") regardless of index.
  2. The Live API receive() generator should iterate over all parts in content.parts to ensure no multimodal data or mid-chunk transitions are dropped.

Observed Behavior:

  1. The framework explicitly checks if content.parts[0].function_response, bypassing validation entirely if the text part comes first.
  2. The receive() loop hardcodes content.parts[0], permanently ignoring any subsequent parts sent in the same chunk.

Environment Details:

  • ADK Library Version: main branch
  • Desktop OS: N/A (Architecture bug)
  • Python Version: 3.11+

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-3.1-flash-live-preview

🟡 Optional Information

Regression:
N/A - This appears to be a systemic issue tied to early architectural assumptions that chunks and payloads would only ever contain a single part.

Logs:

N/A - Issue prevents proper error logs from being emitted due to silent data drops/misrouting.

Screenshots / Video:
N/A

Additional Context:
While it might be rare in current API responses for certain models to stream multiplexed chunks, the Gemini protocol does not prohibit it. The API could technically stream multiplexed chunks (e.g., an image with text, or multiple tool responses at once). By removing these fragile parts[0] assumptions, the framework becomes robust against future Live API updates.

Minimal Reproduction Code:

import pytest
from google.genai import types

# Example of the Mixed Content Validation Bypass
@pytest.mark.asyncio
async def test_mixed_content_bypass(gemini_connection):
    function_response = types.FunctionResponse(name='test_function', response={'result': 'success'})
    content = types.Content(
        role='user',
        parts=[
            types.Part.from_text(text='Hello'),
            types.Part(function_response=function_response),
        ],
    )

    # EXPECTED: ValueError("Function-response content cannot mix...")
    # OBSERVED: Silently misroutes to standard send() pipeline.
    await gemini_connection.send_content(content)

How often has this issue occurred?:

  • Always (100%) - Occurs reliably whenever the framework encounters a multi-part array where the target data is not at index 0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions