[Bug]: hosted_vllm: merge_reasoning_content_in_choices: unsupported operand type(s) for +=: 'NoneType' and 'str'

[vividfog]

What happened?

Hosting QwQ-32B on latest vLLM: vllm serve Qwen/QwQ-32B ... --enable-reasoning --reasoning-parser deepseek_r1

As expected, there is a reasoning_content block and a content block in the response. The model works well like this, if I don't try to merge_reasoning_content_in_choices: true in litellm_params:

{
  "id": "chatcmpl-349de6c1a99841ca916e472d0f9f63da",
  "created": 1743075607,
  "model": "hosted_vllm/Qwen/QwQ-32B",
  "object": "chat.completion",
  "system_fingerprint": null,
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "\n\nHello! How can I assist you today? Feel free to ask me any questions or let me know if you need help with a specific task! 😊",
        "role": "assistant",
        "tool_calls": null,
        "function_call": null,
        "reasoning_content": "Okay, the user just said \"HI\". I need to respond in a friendly way. Let me start by greeting them back. Maybe say \"Hello!\" to keep it simple and open. Then ask how I can assist them today. That should encourage them to share what they need help with. I should keep it welcoming and not too formal. Let me check if there's anything else I should consider. Oh right, the user might be starting a conversation, so I should make sure to invite them to ask questions or share tasks. Yeah, that sounds good. Alright, I'll go with that.\n"
      }
    }
  ],
  "usage": {
    "completion_tokens": 155,
    "prompt_tokens": 11,
    "total_tokens": 166,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  },
  "service_tier": null,
  "prompt_logprobs": null
}

But if I set merge_reasoning_content_in_choices: true in config.yaml for QwQ, there is an error.

  - model_name: Qwen/QwQ-32B-thinking
    litellm_params:
      model: "hosted_vllm/Qwen/QwQ-32B"
      api_base: http://VLLM_IP_ADDRESS:8000/v1
      api_key: none
      temperature: 0.6
      top_k: 40
      top_p: 0.95
      merge_reasoning_content_in_choices: true

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIException - unsupported operand type(s) for +=: 'NoneType' and 'str'

And the reasoning content does not get converted into a stream or block between <think> and </think> tags. I also tested with openai/ as the model provider, and the error is the same. Details attached. I believe the error would be the same with any vLLM-hosted model that can produce reasoning_content.

Expected behavior:

Reasoning content and regular content is merged, with reasoning placed between <think>\n and </think>\n tags, in streaming and non-streaming modes.

I saw that #9029 handles this for Bedrock, Anthropic, Deepseek. Maybe the openai and hosted_vllm providers need the same logic added to them?

Helpful docs to update:

• https://docs.litellm.ai/docs/tutorials/openweb_ui
• https://docs.litellm.ai/docs/providers/vllm

Relevant log output

11:36:24 - LiteLLM Proxy:ERROR: proxy_server.py:3082 - litellm.proxy.proxy_server.async_data_generator(): Exception occured - litellm.APIConnectionError: APIConnectionError: OpenAIException - unsupported operand type(s) for +=: 'NoneType' and 'str'
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1443, in chunk_creator
    return self.return_processed_chunk_logic(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        completion_obj=completion_obj,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        model_response=model_response,  # type: ignore
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        response_obj=response_obj,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 887, in return_processed_chunk_logic
    self._optional_combine_thinking_block_in_choices(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model_response=model_response
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 966, in _optional_combine_thinking_block_in_choices
    model_response.choices[0].delta.content += (
        "<think>" + reasoning_content
    )
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 3057, in async_data_generator
    async for chunk in proxy_logging_obj.async_post_call_streaming_iterator_hook(
    ...<18 lines>...
            yield f"data: {str(e)}\n\n"
  File "/usr/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 270, in async_post_call_streaming_iterator_hook
    async for item in response:
        yield item
  File "/usr/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 270, in async_post_call_streaming_iterator_hook
    async for item in response:
        yield item
  File "/usr/lib/python3.13/site-packages/litellm/integrations/custom_logger.py", line 270, in async_post_call_streaming_iterator_hook
    async for item in response:
        yield item
  [Previous line repeated 3 more times]
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1857, in __anext__
    raise exception_type(
    ...<5 lines>...
    )
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1714, in __anext__
    processed_chunk: Optional[ModelResponseStream] = self.chunk_creator(
                                                     ~~~~~~~~~~~~~~~~~~^
        chunk=chunk
        ^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1454, in chunk_creator
    raise exception_type(
          ~~~~~~~~~~~~~~^
        model=self.model,
        ^^^^^^^^^^^^^^^^^
        custom_llm_provider=self.custom_llm_provider,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        original_exception=e,
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2219, in exception_type
    raise e  # it's already mapped
    ^^^^^^^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 466, in exception_type
    raise APIConnectionError(
    ...<7 lines>...
    )
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIException - unsupported operand type(s) for +=: 'NoneType' and 'str'

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

ghcr.io/berriai/litellm:latest, ghcr.io/berriai/litellm-database:main-latest

Twitter / LinkedIn details

No response

[pedometer77]

litellm/litellm/litellm_core_utils/streaming_handler.py

Line 923 in b210639

model_response.choices[0].delta.content += (

Instead of +=, = should be used

[ziozzang]

bump. me also.

[TechBeme]

bump

[ufwtlsb]

vllm return content and reasoning_content in same package json

[bariccattion]

bump

[dafeliton]

This is fixed as of v1.75.5 (and likely 1.74.15 too)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.