Model communication through LiteLLM API Gateway

[santoshbt]

I have a custom LiteLLM API gateway, which serves the model output. Basically, I cannot mention any model explicitly, instead it has to pass through my api gateway. But if I don't mention model, it by default takes Claude 4 Sonnet, which I don't want. Is there any solution for my requirement?

[Manoj-Gujare]

Hi @santoshbt ,

Why it defaults to Claude Sonnet: Strands uses Amazon Bedrock with a default Claude Sonnet model only when no model provider is passed to the Agent. There is no "leave the model unset and route through my gateway" mode — an Agent() with no model= always falls back to the Bedrock default. The fix is to construct a model provider that points at your LiteLLM gateway and pass it explicitly via Agent(model=...). Once you do, the Bedrock/Claude default never triggers.

About "I can't mention a model": Your LiteLLM gateway is an OpenAI-compatible endpoint, and that API still requires a model field on every request — but it's your gateway's virtual model name / alias, which the proxy maps to a real backend. So you don't name a foundation model in your code; you name the gateway alias. If you want the gateway to fully decide, define one alias (or a router) on the proxy and use it everywhere.

Option A — OpenAIModel pointed at the gateway (recommended)

LiteLLM Proxy behaves as an OpenAI-compatible API, so Strands' OpenAI provider is the least-friction fit:

from strands import Agent
from strands.models.openai import OpenAIModel

model = OpenAIModel(
    client_args={
        "base_url": "https://your-litellm-gateway/v1",  # your gateway (drop /v1 if it's mounted at root)
        "api_key": "<your-gateway-virtual-key>",
    },
    model_id="<alias-your-gateway-exposes>",  # gateway routing alias, NOT a real model name
    params={"temperature": 0.7, "max_tokens": 1024},
)

agent = Agent(model=model)

• base_url inside client_args is the documented way to target any OpenAI-compatible server.
• model_id is required (the OpenAI client always sends model); set it to your gateway's alias. If your gateway ignores model and routes purely by the virtual key, any value the gateway accepts works.
• Install: pip install 'strands-agents[openai]'

Option B — LiteLLMModel pointed at the proxy

If you prefer the LiteLLM provider, you must explicitly tell LiteLLM to treat the base URL as a proxy; otherwise it tries to call the upstream provider directly:

import litellm
from strands import Agent
from strands.models.litellm import LiteLLMModel

litellm.use_litellm_proxy = True   # required so base_url is treated as a proxy

model = LiteLLMModel(
    client_args={
        "base_url": "https://your-litellm-gateway",
        "api_key": "<your-gateway-virtual-key>",
    },
    model_id="<alias>",   # or prefix-route with "litellm_proxy/<alias>"
)

agent = Agent(model=model)

• Setting base_url alone is not enough. Without litellm.use_litellm_proxy = True (or the litellm_proxy/ model prefix), LiteLLM routes directly to the upstream provider instead of your gateway. (See [sdk-python issue #661]([BUG] LiteLLMModel does not properly use custom proxy server settings #661).)
• Install: pip install 'strands-agents[litellm]'

Bottom line

• The Claude-4-Sonnet default appears only because no model is configured; pass an explicit gateway-backed provider and it's gone.
• You still pass a model_id, but it's your gateway alias, not a foundation-model name — the gateway resolves it. Define a single alias if you want the gateway to be the one source of truth.
• Either OpenAIModel (treat the proxy as OpenAI-compatible) or LiteLLMModel (with use_litellm_proxy=True) works; OpenAIModel is usually the simplest.

References

• OpenAI / OpenAI-compatible provider: https://strandsagents.com/docs/user-guide/concepts/model-providers/openai/
• LiteLLM provider: https://strandsagents.com/docs/user-guide/concepts/model-providers/litellm/
• LiteLLM proxy routing behavior: [BUG] LiteLLMModel does not properly use custom proxy server settings #661