[Bug]: LiteLLM Proxy Incorrectly Sets max_tokens to 4096 for Claude 3.5/3.7

[ljk53]

What happened?

For Anthropic Claude 3.5 and 3.7 models, the default maximum output token count should be 8192. However, when sending a request without the max_tokens parameter to the LiteLLM proxy, it adds a max_tokens parameter with the default value 4096, which is incorrect for Claude 3.5/3.7.

Send a request to the LiteLLM proxy without specifying max_tokens:

curl -X POST \
http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "claude-3-7-sonnet-20250219", "messages": [{"role": "system", "content": "you are a helpful AI assistant"}, {"role": "user", "content": [{"type": "text", "text": "How to write binary search in python?"}]}]}'

Observe the request that LiteLLM sends to the Anthropic API:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/messages \
-H 'anthropic-version: *****' -H 'x-api-key: sk-********************************************' -H 'accept: *****' -H 'content-type: *****' \
-d '{'model': 'claude-3-7-sonnet-20250219', 'messages': [{'role': 'user', 'content': [{'type': 'text', 'text': 'How to write binary search in python?'}]}], 'system': [{'type': 'text', 'text': 'you are a helpful AI assistant'}], 'max_tokens': 4096}'

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.61.13-stable

Twitter / LinkedIn details

No response

[ishaan-jaff]

the default maximum output token count should be 8192

Hi @ljk53 , why should this be the default ?

[ljk53]

@ishaan-jaff because Anthropic requires a max_tokens parameter — unlike some other LLM providers — the current implementation of LiteLLM proxy automatically assigns a default value (4096) if the caller does not explicitly specify it. While 4096 was appropriate prior to the release of Claude 3.5, the newer models have increased this max_token limit to 8192.
As a LiteLLM caller, the expectation is to seamlessly switch between various LLM providers using sensible defaults that fully leverage each provider's capabilities. Therefore, the default max_tokens value should now reflect Claude 3.5's enhanced capacity. Alternatively, it should somehow warns callers to set the max_tokens parameter explicitly, preventing unexpected limitations from outdated default values (it's a little bit subtle to find out as it does not always generates long output).

[sammcj]

Came to log this as well, found that the outputs from Sonnet 3.7 were getting artificially limited at 4k which is half the default of 8k.

[AlessandroMondin]

@ljk53 did it fix it for you passing the max_tokens parameter?
I am testing fireworks models, and the value that I pass to max_tokens is ignored.

[retanoj]

I think you can config your model provider in litellm by setting max_tokens field to 8192, which means litellm proxy will set the max_tokens value to 8192 when max_tokens not appears in the original request.

[wojciech-mikolajczyk-yohana]

Are there any plans or timeline of fixing that? The behavior of that currently is a bit confusing