[Bug]: Vertex Gemini web search streaming crashes on 3/3.1 Flash / Flash Lite models with empty choices chunk

[takumamitarai]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

What happened

When using Vertex AI Gemini with stream=True and web_search_options={}, LiteLLM fails mid-stream with:

litellm.MidStreamFallbackError: litellm.APIConnectionError: list index out of range

Confirmed failing models:

vertex_ai/gemini-3.1-flash-lite-preview
vertex_ai/gemini-3.1-flash-lite
vertex_ai/gemini-3-flash-preview

The same request pattern worked with:

vertex_ai/gemini-3.1-pro-preview

So this appears to affect Gemini Flash / Flash Lite streaming response shapes with web search, rather than all Gemini 3.x models.

Root cause hypothesis

The failure appears to happen inside LiteLLM's streaming pipeline.

Call flow:

litellm.main.acompletion()
  -> litellm.main.completion()
  -> litellm.utils.get_optional_params()
  -> VertexGeminiConfig.map_openai_params()
  -> VertexGeminiConfig._map_web_search_options()
  -> VertexLLM.completion()
  -> VertexLLM.async_streaming()
  -> make_call()
  -> ModelResponseIterator.chunk_parser()
  -> CustomStreamWrapper.chunk_creator()
  -> CustomStreamWrapper.return_processed_chunk_logic()
  -> CustomStreamWrapper.raise_on_model_repetition()

web_search_options={} is converted by:

VertexGeminiConfig._map_web_search_options()

into a Gemini googleSearch tool.

During streaming, ModelResponseIterator.chunk_parser() creates a stream response object with empty choices first:

model_response = ModelResponseStream(choices=[], id=response_id)

and only populates choices when candidate content is present.

For Gemini Flash / Flash Lite models with web search streaming, some metadata-only chunks appear to produce a ModelResponseStream with empty choices.

Later, CustomStreamWrapper.raise_on_model_repetition() assumes choices are non-empty:

last_content = self.chunks[-1].choices[0].delta.content
second_to_last_content = self.chunks[-2].choices[0].delta.content

This raises:

IndexError: list index out of range

LiteLLM then wraps it as APIConnectionError, then MidStreamFallbackError.

Expected behavior

LiteLLM should not fail when a Vertex Gemini streaming chunk has no choices.

Possible expected behavior:

• skip empty-choice metadata-only chunks, or
• guard raise_on_model_repetition() against chunks with empty choices, or
• ensure ModelResponseIterator.chunk_parser() returns None instead of ModelResponseStream(choices=[]) when the chunk has no streamable delta.

Actual behavior

LiteLLM raises mid-stream:

MidStreamFallbackError -> APIConnectionError -> IndexError: list index out of range

Environment

LiteLLM version: 1.83.14
Python: 3.12
Provider: Vertex AI
Endpoint style: Vertex AI streaming / streamGenerateContent

Failing models:
  - vertex_ai/gemini-3.1-flash-lite-preview
  - vertex_ai/gemini-3.1-flash-lite
  - vertex_ai/gemini-3-flash-preview

Working comparison model:
  - vertex_ai/gemini-3.1-pro-preview

I also checked current LiteLLM main around version 1.85.0, and the relevant code still appears to assume non-empty choices in raise_on_model_repetition():

last_content = self.chunks[-1].choices[0].delta.content
second_to_last_content = self.chunks[-2].choices[0].delta.content

Related issues

Possibly related, but not the same failure:

• [Bug]: Gemini grounding silently disabled when web_grounding tools are mixed with function tools (3 compounding failures) #27479
• [Bug]: stream_chunk_builder corrupts Gemini server_side_tool_invocations and thought_signatures in streaming, causing "Corrupted tool call context" on follow-up turns #25869
• [Bug]: Gemini gemini-2.5-flash-preview-05-20 stream mode concatenates reasoning data with main content due to missing thought/reasoning chunk identification #11000

### Steps to Reproduce

```python
async def main() -> None:
    vertex_credentials = os.environ.get("GOOGLE_VERTEX_AI_CREDENTIALS")
    vertex_project = os.environ.get("GOOGLE_VERTEX_AI_PROJECT")

    for model in MODELS:
        print(f"\nTesting {model}")
        response = await acompletion(
            model=model,
            messages=[
                {
                    "role": "user",
                    "content": "Search the web and summarize the latest news about OpenAI.",
                }
            ],
            stream=True,
            web_search_options={},
            stream_options={"include_usage": True},
            vertex_credentials=vertex_credentials,
            vertex_project=vertex_project,
            vertex_location="global",
        )

        async for chunk in response:
            print(chunk)


if __name__ == "__main__":
    asyncio.run(main())

Relevant log output

File ".../litellm/litellm_core_utils/streaming_handler.py", line 1587, in chunk_creator
  return self.return_processed_chunk_logic(...)

File ".../litellm/litellm_core_utils/streaming_handler.py", line 960, in return_processed_chunk_logic
  self.raise_on_model_repetition()

File ".../litellm/litellm_core_utils/streaming_handler.py", line 272, in raise_on_model_repetition
  last_content = self.chunks[-1].choices[0].delta.content
                 ~~~~~~~~~~~~~~~~~~~~~~~^^^

IndexError: list index out of range

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

1.83.14

Twitter / LinkedIn details

@secchinman

[rehan243]

LLM engineering has a lot of nuance! Some production learnings:

• Model selection: We benchmark every new model on our specific tasks before
  switching. Leaderboard performance doesn't always translate to production use cases
• Prompt versioning: We version-control prompts like code and run A/B tests
  when making changes — small prompt tweaks can have outsized effects
• Fallback chains: We route between providers (OpenAI -> Anthropic -> self-hosted)
  with automatic failover. No single point of failure

The pace of improvement is incredible — staying current with new models and techniques
is almost a full-time job!

[Divyansh8321]

Hey! I'd like to work on this. The root cause is that raise_on_model_repetition() in streaming_handler.py unconditionally accesses choices[0] on historical chunks (lines 272 and 282), assuming choices is always non-empty. Vertex Gemini Flash/Flash Lite sends metadata-only chunks during web search streaming (grounding data, URL context) that produce ModelResponseStream(choices=[]) — unlike Gemini Pro which doesn't emit these intermediate chunks.

Fix: add a guard after the existing len(self.chunks) < 2 check to return early if either of the last two chunks has empty choices. This follows the same pattern already used elsewhere in the file (e.g. line 870). Can this be assigned to me?

[rehan243]

Yeah, that version issue with Vertex Gemini Flash is a pain. That makes sense, we've seen similar issues with empty chunks causing crashes. Good catch on the pattern at line 870.

I'd say go ahead and add that guard. It's a solid fix. Just make sure to add a test case for this edge case.

You're on it, right? Let me know if that helps.

[sandy1990418]

I’m seeing a similar issue as well, but with vertex_ai/gemini-3.5-flash.

[tobiasmuecksch]

I`m experiencing this issue with google gemini 3.1 pro and with anthropic claude sonnet 4.6 also.
This is a really painful issue to me 😢

[JuampiHernandez]

Fixed in #30485. Gemini Flash and Lite web-search streams can include metadata-only chunks with choices: []. LiteLLM now skips repetition detection for those chunks instead of crashing mid-stream.\n\nCurrent main has regression coverage. Closing as resolved.