[Bug]: GroqException - list index out of range

[ChenghaoMou]

What happened?

Not sure if this is a Groq problem or litellm problem. Here is a code snippet to reproduce the issue:

import os

import litellm

os.environ["LITELLM_LOG"] = "DEBUG"


async def main():
    response = await litellm.acompletion(
        model="groq/llama-3.3-70b-specdec",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": [{"type": "text", "text": "Test"}],
            },
        ],
        stream=True,
        stream_options={"include_usage": True},
    )
    async for chunk in response:
        print(chunk)


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

Turning off include_usage will make it work. From the log, it looks like it is using databricks for parsing.

Relevant log output

uv run --with "litellm==1.61.7"  temp.py

ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='It', role='assistant', function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' looks', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' like', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' you', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content="'re", role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' just', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' testing', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' the', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' waters', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='.', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' Is', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' there', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' something', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' I', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' can', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' help', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' you', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' with', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' or', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' would', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' you', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' like', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' to', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' start', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' a', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' conversation', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='?', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' I', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content="'m", role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' here', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' to', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' assist', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' you', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' with', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' any', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' questions', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' or', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' topics', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' you', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content="'d", role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' like', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' to', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' discuss', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='.', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145122, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-b4e476ff-5fde-456e-9b86-0549c2652542', created=1740145123, model='llama-3.3-70b-specdec', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], stream_options={'include_usage': True}, usage=Usage(completion_tokens=44, prompt_tokens=6, total_tokens=50, completion_tokens_details=None, prompt_tokens_details=None))

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.


Provider List: https://docs.litellm.ai/docs/providers

Traceback (most recent call last):
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1548, in __anext__
    async for chunk in self.completion_stream:
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/llms/databricks/streaming_utils.py", line 143, in __anext__
    return self.chunk_parser(chunk=json_chunk)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/llms/databricks/streaming_utils.py", line 28, in chunk_parser
    if processed_chunk.choices[0].delta.content is not None:  # type: ignore
       ~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chenghao/Developer/ai-worker/temp.py", line 28, in <module>
    asyncio.run(main())
  File "/Users/chenghao/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/chenghao/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-worker/temp.py", line 21, in main
    async for chunk in response:
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1703, in __anext__
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2207, in exception_type
    raise e  # it's already mapped
    ^^^^^^^
  File "/Users/chenghao/Developer/ai-worker/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 463, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: GroqException - list index out of range

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

1.61.7

Twitter / LinkedIn details

No response

[wwwillchen]

Getting a similar exception as above:

I don't think this is a Groq error because I've tried calling Groq directly (using the openAI client) and it works and with the exact same input it's getting the error thrown.

Weirdly though, the streaming response prior to the error looks OK (i.e. the content returned before the error is the same as calling Groq directly).

Thanks in advance!

23:24:34 - LiteLLM Proxy:ERROR: proxy_server.py:2995 - litellm.proxy.proxy_server.async_data_generator(): Exception occured - litellm.APIConnectionError: APIConnectionError: GroqException - list index out of range

Traceback (most recent call last):

  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1545, in __anext__

    async for chunk in self.completion_stream:

    ...<50 lines>...

        return processed_chunk

  File "/usr/lib/python3.13/site-packages/litellm/llms/databricks/streaming_utils.py", line 143, in __anext__

    return self.chunk_parser(chunk=json_chunk)

           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.13/site-packages/litellm/llms/databricks/streaming_utils.py", line 28, in chunk_parser

    if processed_chunk.choices[0].delta.content is not None:  # type: ignore

       ~~~~~~~~~~~~~~~~~~~~~~~^^^

IndexError: list index out of range

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 2974, in async_data_generator

    async for chunk in response:

    ...<14 lines>...

            yield f"data: {str(e)}\n\n"

  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1700, in __anext__

    raise exception_type(

          ~~~~~~~~~~~~~~^

        model=self.model,

        ^^^^^^^^^^^^^^^^^

    ...<3 lines>...

        extra_kwargs={},

        ^^^^^^^^^^^^^^^^

    )

    ^

  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2195, in exception_type

    raise e  # it's already mapped

    ^^^^^^^

  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 461, in exception_type

    raise APIConnectionError(

    ...<7 lines>...

    )

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: GroqException - list index out of range

[salvador-radpair]

I was also getting a similar error using LiteLLM, but just recently started getting this error using the Groq client, no LiteLLM.

[loorisr]

I have the same using the openAI client with LiteLLM as a proxy.
it seems to come from stream_options={"include_usage": True}

[yigitkonur]

Same here too - any planned patch @ishaan-jaff

[azdolinski]

I set as an extra option (directly in postgrade database LiteLLM_ProxyModelTable... i don't see this option on UI)

and it is working now

so try to set for model

litellm_params:
  stream_options: {}

[bachya]

Ran into this, as well. This script (with debug turned on):

import asyncio

import litellm

litellm._turn_on_debug()


async def main():
    response = await litellm.acompletion(
        "groq/llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": "Write a haiku about redheads."}],
        api_key="REDACTED",
        stream=True,
        stream_options={"include_usage": True},
    )
    async for chunk in response:
        print(chunk)


asyncio.run(main())

...returns this:

19:00:37 - LiteLLM:DEBUG: utils.py:311 -

19:00:37 - LiteLLM:DEBUG: utils.py:311 - Request to litellm:
19:00:37 - LiteLLM:DEBUG: utils.py:311 - litellm.acompletion('groq/llama-3.3-70b-versatile', messages=[{'role': 'user', 'content': 'Write a haiku about redheads.'}], api_key='REDACTED', stream=True, stream_options={'include_usage': True})
19:00:37 - LiteLLM:DEBUG: utils.py:311 -

19:00:37 - LiteLLM:DEBUG: litellm_logging.py:389 - self.optional_params: {}
19:00:37 - LiteLLM:DEBUG: utils.py:311 - ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
19:00:37 - LiteLLM:DEBUG: caching_handler.py:212 - CACHE RESULT: None
19:00:37 - LiteLLM:INFO: utils.py:2999 -
LiteLLM completion() model= llama-3.3-70b-versatile; provider = groq
19:00:37 - LiteLLM:DEBUG: utils.py:3002 -
LiteLLM: Params passed to completion() {'model': 'llama-3.3-70b-versatile', 'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'n': None, 'stream': True, 'stream_options': {'include_usage': True}, 'stop': None, 'max_tokens': None, 'max_completion_tokens': None, 'modalities': None, 'prediction': None, 'audio': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'custom_llm_provider': 'groq', 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': None, 'logprobs': None, 'top_logprobs': None, 'extra_headers': None, 'api_version': None, 'parallel_tool_calls': None, 'drop_params': None, 'reasoning_effort': None, 'additional_drop_params': None, 'messages': [{'role': 'user', 'content': 'Write a haiku about redheads.'}], 'thinking': None}
19:00:37 - LiteLLM:DEBUG: utils.py:3005 -
LiteLLM: Non-Default params passed to completion() {'stream': True, 'stream_options': {'include_usage': True}}
19:00:37 - LiteLLM:DEBUG: utils.py:311 - Final returned optional params: {'stream': True, 'stream_options': {'include_usage': True}, 'extra_body': {}}
19:00:37 - LiteLLM:DEBUG: litellm_logging.py:389 - self.optional_params: {'stream': True, 'stream_options': {'include_usage': True}, 'extra_body': {}}
19:00:37 - LiteLLM:DEBUG: litellm_logging.py:685 -

POST Request Sent from LiteLLM:
curl -X POST \
https://api.groq.com/openai/v1/chat/completions \
-H 'Content-Type: ap****on' -H 'Authorization: Be****nB' \
-d '{'model': 'llama-3.3-70b-versatile', 'messages': [{'role': 'user', 'content': 'Write a haiku about redheads.'}], 'stream_options': {'include_usage': True}, 'stream': True}'


19:00:37 - LiteLLM:DEBUG: logging_utils.py:113 - `logging_obj` not found - unable to track `llm_api_duration_ms
19:00:37 - LiteLLM:DEBUG: utils.py:311 - RAW RESPONSE:
<litellm.llms.databricks.streaming_utils.ModelResponseIterator object at 0x106475160>


19:00:37 - LiteLLM:INFO: cost_calculator.py:588 - selected model name for cost calculation: groq/llama-3.3-70b-versatile
19:00:38 - LiteLLM:DEBUG: utils.py:4295 - checking potential_model_names in litellm.model_cost: {'split_model': 'llama-3.3-70b-versatile', 'combined_model_name': 'groq/llama-3.3-70b-versatile', 'stripped_model_name': 'llama-3.3-70b-versatile', 'combined_stripped_model_name': 'groq/llama-3.3-70b-versatile', 'custom_llm_provider': 'groq'}
19:00:38 - LiteLLM:DEBUG: cost_calculator.py:344 - Returned custom cost for model=groq/llama-3.3-70b-versatile - prompt_tokens_cost_usd_dollar: 0.0, completion_tokens_cost_usd_dollar: 0.0
19:00:38 - LiteLLM:DEBUG: litellm_logging.py:900 - response_cost: 0.0
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='Fi', role='assistant', function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='ery', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' locks', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' abl', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='aze', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='\n', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='G', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='inger', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' souls', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' with', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' hearts', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' ag', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='low', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='\n', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='Summer', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content="'s", role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' warm', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' delight', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})
19:00:38 - LiteLLM:DEBUG: utils.py:311 - token_counter messages received: [{'role': 'user', 'content': 'Write a haiku about redheads.'}]
ModelResponseStream(id='chatcmpl-7f2f47fd-9dca-4d64-88d6-103e8ffedfab', created=1742605238, model='llama-3.3-70b-versatile', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True}, usage=Usage(completion_tokens=18, prompt_tokens=8, total_tokens=26, completion_tokens_details=None, prompt_tokens_details=None))
19:00:38 - LiteLLM:DEBUG: litellm_logging.py:1932 - Logging Details LiteLLM-Failure Call: []

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.


Provider List: https://docs.litellm.ai/docs/providers

19:00:38 - LiteLLM:DEBUG: get_api_base.py:63 - Error occurred in getting api base - litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=llama-3.3-70b-versatile
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers
19:00:38 - LiteLLM:DEBUG: exception_mapping_utils.py:2243 - Logging Details: logger_fn - None | callable(logger_fn) - False
19:00:38 - LiteLLM:DEBUG: utils.py:311 - token_counter messages received: [{'role': 'user', 'content': 'Write a haiku about redheads.'}]
Traceback (most recent call last):
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1671, in __anext__
    async for chunk in self.completion_stream:
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/llms/databricks/streaming_utils.py", line 143, in __anext__
    return self.chunk_parser(chunk=json_chunk)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/llms/databricks/streaming_utils.py", line 28, in chunk_parser
    if processed_chunk.choices[0].delta.content is not None:  # type: ignore
       ~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bachya/Git/liminal/liminal/test.py", line 20, in <module>
    asyncio.run(main())
  File "/Users/bachya/.local/share/mise/installs/python/3.12.8/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/bachya/.local/share/mise/installs/python/3.12.8/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bachya/.local/share/mise/installs/python/3.12.8/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/bachya/Git/liminal/liminal/test.py", line 16, in main
    async for chunk in response:
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1826, in __anext__
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2219, in exception_type
    raise e  # it's already mapped
    ^^^^^^^
  File "/Users/bachya/Git/liminal/liminal/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 466, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: GroqException - list index out of range

[mantrakp04]

bro ur api key

[bgeneto]

I have the same using the openAI client with LiteLLM as a proxy. it seems to come from stream_options={"include_usage": True}

Good catch! It seems to be related to stream_options={"include_usage": True} for me too. Thx for reporting...

[benaduggan]

Is there a way to disable this another way potentially? Trying to use roo code with this and don't think I can disable include_usage on it

[gerasdf]

Me too. And I also have "include_usage": True. I'm almost sure it used to work some time ago.

I run into it today when trying llama4 (groq/meta-llama/lla ama-4-maverick-17b-128e-instruct). It streams all good, until the end.

My understanding is that the last message, which includes cost, doesn't have any choice. Funny is that I have special code to handle no choices, like if it did use to work.

This sort of breaks groq support (at least for some models)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hdnh2006]

I can confirm this is a Groq problem related with the usage parameter.

For example, I have integrated litellm with Open WebUI and I set the following check [ ] Usage

After uncheck that option, any Groq model correctly worked.