[BUG] structured_output with LiteLLM Proxy raises ValueError: Model does not support response_format

[ranajoy-dutta]

Checks

• I have updated to the lastest minor and patch version of Strands
• I have checked the documentation and this is not expected behavior
• I have searched ./issues and there are no duplicates of my issue

Strands Version

1.8.0

Python Version

3.11.13

Operating System

macOS 15.6.1

Installation Method

pip

Steps to Reproduce

When using Agent.structured_output(...) with a LiteLLMModel that points to a LiteLLM proxy, I get the following error:
ValueError: Model does not support response_format
This happens even though the underlying model (Claude 3.7 Sonnet via LiteLLM proxy) supports JSON / structured outputs.

Code to Reproduce

import os
from pydantic import BaseModel, Field
from strands import Agent
from strands.models.litellm import LiteLLMModel

class BookAnalysis(BaseModel):
    """Analyze a book's key information."""
    title: str = Field(description="The book's title")
    author: str = Field(description="The book's author")
    genre: str = Field(description="Primary genre or category")
    summary: str = Field(description="Brief summary of the book")
    rating: int = Field(description="Rating from 1-10", ge=1, le=10)

model = LiteLLMModel(
    client_args={
        "api_key": "DUMMY_API_KEY",         # replaced for repro
        "api_base": "https://<your-litellm-proxy-url>",
        "use_litellm_proxy": True
    },
    model_id="claude-3-7-sonnet-20250219"
)

agent = Agent(model=model)

result = agent.structured_output(
    BookAnalysis,
    """
    Analyze this book: "The Hitchhiker's Guide to the Galaxy" by Douglas Adams.
    It's a science fiction comedy about Arthur Dent's adventures through space
    after Earth is destroyed. It's widely considered a classic of humorous sci-fi.
    """
)

Expected Behavior

• Agent.structured_output(...) should pass the Pydantic model schema to LiteLLM as response_format.
• LiteLLM proxy should return JSON that conforms to the schema.
• The response should be parsed back into the provided Pydantic model.

Actual Behavior

Traceback

  File "example.py", line 30, in <module>
    result = agent.structured_output(BookAnalysis, "Analyze this book ...")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ...
  File ".../strands/models/litellm.py", line 215, in structured_output
    raise ValueError("Model does not support response_format")
ValueError: Model does not support response_format

Additional Context

• Works fine when calling litellm.completion(..., response_format=BookAnalysis) directly.
• Works fine with strands-Litellm (without proxy)
• Fails only when wrapped inside Agent.structured_output(...) with LiteLLMModel.
• Possibly related to how LiteLLMModel checks response_format support.

Possible Solution

No response

Related Issues

No response

[dbschmigelski]

Hi so we have a check using

from litellm.utils import supports_response_schema

if not supports_response_schema(self.get_config()["model_id"]):

I agree that this seems like a case where we did not consider the proxy case. We'll need to see if we can reliable determine if a proxy model supports it. There is also a world where we simply accept that we cannot determine this, and if a proxy is being used we just skip the if statement.

Thanks for raising this.

[ranajoy-dutta]

@dbschmigelski thanks for taking a look into this.
I think, it would be great to have some sort of override using flag, that user can pass to bypass the check. Or, yes simply skip the if statement if it's a proxy and let user handle that scenario in application.
Eitherways, is there any plan on having this fixed soon?

[dbschmigelski]

@ranajoy-dutta, this is something I am keen on fixing quickly. Can you provide some additional info though.

Regarding claude-3-7-sonnet-20250219, are you attempting to go through anthropic here or bedrock?

[dbschmigelski]

Hi, so here is some background.

Some providers support natively structured_output.

OpenAI for example does and you can see this in our OpenAIModel implementation where we leverage the native response_format feature.

Other providers, like Bedrock and Anthropic, do NOT support structured output. In these cases we get a little clever by doing the structuring ourselves (Bedrock, Anthropic).

For LiteLLM we should not simply fail hard. We should leverage native capabilities, if available. If not, Strands should employ the same strategy used in BedrockModel and AnthropicModel. Basically, customers should never simple fail to use structured output.

There may also a bug in LiteLLM for this input we see that there is proxy support here https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/get_llm_provider_logic.py#L88 but the actual call never passes this information https://github.com/BerriAI/litellm/blob/f34bbd1f35d4e3fea4f8b8b2409b6018155d10c9/litellm/utils.py#L1986

We will be creating an issue in LiteLLM to fix this if needed

[ranajoy-dutta]

I'm using bedrock.
Yeah, agreed with this.

Hi, so here is some background.

Some providers support natively structured_output.

OpenAI for example does and you can see this in our OpenAIModel implementation where we leverage the native response_format feature.

Other providers, like Bedrock and Anthropic, do NOT support structured output. In these cases we get a little clever by doing the structuring ourselves (Bedrock, Anthropic).

For LiteLLM we should not simply fail fast. We should leverage native capabilities, if available. If not, Strands should employ the same strategy used in BedrockModel and AnthropicModel. Basically, customers should never simple fail to use structured output.

There may also a bug in LiteLLM for this input we see that there is proxy support here https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/get_llm_provider_logic.py#L88 but the actual call never passes this information https://github.com/BerriAI/litellm/blob/f34bbd1f35d4e3fea4f8b8b2409b6018155d10c9/litellm/utils.py#L1986

We will be creating an issue in LiteLLM to fix this if needed

[Ratish1]

I would like to contribute to this

[ranajoy-dutta]

Following up on this... @dbschmigelski saw that you are working on this. Let me know if you'd like any assistance or collaboration in some way.