[Bug]: LiteLLM Router carries over completion parameters across requests

[mserek]

What happened?

We noticed that when using instructor with litellm, arguments to completion are passed over to subsequent requests, if they are not provided in the next request (example below). The problem seems to stem from router.chat.completion.create, the same issue can be easily illustrated by calling router with router.chat.completion.create instead of completion, passing max_tokens, and then trying to call router.embedding which will fail.

If this is expected, then how could we use Router with instructor properly? In the docs, examples I've seen call it with router.chat.completion.create.

Example 1:

import instructor
from litellm import Router
from pydantic import BaseModel
import litellm

litellm._turn_on_debug()


def test_instructor_carries_over_args() -> None:
    client = instructor.patch(
        Router(
            model_list=[
                {
                    "model_name": "gpt-4o-mini",
                    "litellm_params": {
                        "model": "azure/gpt-4o-mini",
                        "api_key": AZURE_OPENAI_API_KEY,
                        "api_version": "2024-10-21",
                        "api_base": AZURE_OPENAI_API_BASE,
                    },
                }
            ],
        )
    )

    class SomeModelResponse(BaseModel):
        joke: str

    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": "Write a simple joke."}],
        response_model=SomeModelResponse,
    )
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": "Write a simple joke."}],
        max_tokens=128,
        temperature=0.0,
        top_p=0.0,
        timeout=2,
        response_model=SomeModelResponse,
    )
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Write a joke.",
            }
        ],
        response_model=SomeModelResponse,
    )
test_instructor_carries_over_args()

# First request with no additional params (as expected)
# POST Request Sent from LiteLLM:

# -d '{'model': 'gpt-4o-mini', 'messages': [{'role': 'system', 'content': 'Write a simple joke.'}], 'tools': [{'type': 'function', 'function': {'name': 'SomeModelResponse', 'description': 'Correctly extracted `SomeModelResponse` with all the required parameters with correct types', 'parameters': {'properties': {'joke': {'title': 'Joke', 'type': 'string'}}, 'required': ['joke'], 'type': 'object'}}}], 'tool_choice': {'type': 'function', 'function': {'name': 'SomeModelResponse'}}, 'extra_body': {}}'

# Second request with additional params (as expected)
# POST Request Sent from LiteLLM:

# -d '{'model': 'gpt-4o-mini', 'messages': [{'role': 'system', 'content': 'Write a simple joke.'}], 'temperature': 0.0, 'top_p': 0.0, 'max_tokens': 128, 'tools': [{'type': 'function', 'function': {'name': 'SomeModelResponse', 'description': 'Correctly extracted `SomeModelResponse` with all the required parameters with correct types', 'parameters': {'properties': {'joke': {'title': 'Joke', 'type': 'string'}}, 'required': ['joke'], 'type': 'object'}}}], 'tool_choice': {'type': 'function', 'function': {'name': 'SomeModelResponse'}}, 'extra_body': {}}'

# Third request (Unexpectedly has 'temperature': 0.0, 'top_p': 0.0, 'max_tokens': 128)
# POST Request Sent from LiteLLM:

# -d '{'model': 'gpt-4o-mini', 'messages': [{'role': 'system', 'content': 'Write a joke.'}], 'temperature': 0.0, 'top_p': 0.0, 'max_tokens': 128, 'tools': [{'type': 'function', 'function': {'name': 'SomeModelResponse', 'description': 'Correctly extracted `SomeModelResponse` with all the required parameters with correct types', 'parameters': {'properties': {'joke': {'title': 'Joke', 'type': 'string'}}, 'required': ['joke'], 'type': 'object'}}}], 'tool_choice': {'type': 'function', 'function': {'name': 'SomeModelResponse'}}, 'extra_body': {}}'

Example 2:

from litellm import Router

def test_error_on_embeddings():
    router = Router(
        model_list=[
            {
                "model_name": "gpt-4o-mini",
                "litellm_params": {
                    "model": "azure/gpt-4o-mini",
                    "api_key": AZURE_OPENAI_API_KEY,
                    "api_version": "2024-10-21",
                    "api_base": AZURE_OPENAI_API_BASE,
                }},
                {
                    "model_name": "ada-embeddings",
                    "litellm_params": {
                        "model": "azure/text-embedding-ada-002",
                        "api_key": AZURE_OPENAI_API_KEY,
                        "api_version": "2024-10-21",
                        "api_base": AZURE_OPENAI_API_BASE,
                    },
                }
            ],
        )
    router.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello."}],
        max_tokens=128,
    )
    router.embedding(
        model="ada-embeddings",
        input=["Hello."],
    )

test_error_on_embeddings()

# Errors with litellm.exceptions.APIError: litellm.APIError: AzureException APIError - Embeddings.create() got an unexpected keyword argument 'max_tokens'. Received Model Group=ada-embeddings
Available Model Group Fallbacks=None LiteLLM Retried: 1 times, LiteLLM Max Retries: 2

Relevant log output

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.65.4

Twitter / LinkedIn details

No response

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.