[Bug]: 500 error when embedding

[ncecere]

What happened?

after a few minutes when trying to use the embedding endpoint I 500 error

I am on version v1.35.38-stable, and this happens with any embedding model that I use.

Relevant log output

`{"error":{"message":"AsyncEmbeddings.create() got an unexpected keyword argument 'messages'","type":"None","param":"None","code":500}}`

Twitter / LinkedIn details

No response

[ncecere]

when the issue you starts I can't find anything in the litellm logs, If you have somewhere I should look please let me know and I will. the response/error is from me running a curl command

[ishaan-jaff]

Does this occur in 1.35.38 @ncecere (non-stable tag)?

[ncecere]

yes it does @ishaan-jaff , it happens in the non stable tag too. I have tried every version from 1.35.36 - 1.35.38

[ncecere]

I am adding the model using this pattern

- model_name: SFR-Embedding-Mistral
    litellm_params: 
      model: openai/Salesforce/SFR-Embedding-Mistral
      api_key: sk-1234
      api_base: http://server.example.com/
    model_info:
      mode: embedding

[ncecere]

example of the command I'm running

curl https://api.server.example.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-123" \
  -d '{
    "input": "Your text here",
    "model": "sfr-embedding-mistral"
  }'

[ncecere]

Just to add some more context to this, when running version v1.35.8 everything is fine. I have tried going all the way back to version v1.35.15-stable and I still see the same problem with embeddings. it will work for a while and then I start getting the 500 error. I am using the infinity project to run my embedding models. If you need anymore information from me please let me know.

[krrishdholakia]

A AsyncEmbeddings.create() got an unexpected keyword argument 'messages'"

This looks like an error from your custom endpoint @ncecere

Can you share what the call it receives looks like?

[ncecere]

so the issue seems to be that extra litellm params are getting set when I add the embedding models.

[ncecere]

I'm adding the model via the rest API but I'm not setting the messages part. is there a special method I should use when adding/creating and embedding model via the api?

[ncecere]

{"model": "openai/Salesforce/SFR-Embedding-Mistral", "api_base": "http://dgx-s29.ai.apps.it.ufl.edu:8007/", "extra_body": {"cache": {"no-cache": True}}, "input_cost_per_token": 1e-09, "output_cost_per_token": 1e-08, "messages": [{"role": "user", "content": "What"s 1 + 1?"}], "input": "Your text herhhe", "caching": True, "client": <openai.AsyncOpenAI object at 0x7f14a83499d0>, "proxy_server_request": {"url": "https://api.ai.it.ufl.edu/v1/embeddings", "method": "POST", "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "authorization": "Bearer sk-REDACTED", "content-length": "75"}, "body": {"input": "Your text herhhe", "model": "sfr-embedding-mistral"}}, "user": "default_user_id", "metadata": {"user_api_key": "REDACTED", "user_api_key_metadata": {}, "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "content-length": "75"}, "user_api_key_alias": None, "user_api_key_user_id": "default_user_id", "user_api_key_team_id": None, "user_api_key_team_alias": None, "endpoint": "https://api.ai.it.ufl.edu/v1/embeddings", "model_group": "sfr-embedding-mistral", "deployment": "openai/Salesforce/SFR-Embedding-Mistral", "model_info": {"id": "55f72312-eb20-7afb-e813-667a9d374e17", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "api_base": "http://dgx-s29.ai.apps.it.ufl.edu:8007/", "caching_groups": None, "previous_models": [{"exception_type": "TypeError", "exception_string": "AsyncEmbeddings.create() got an unexpected keyword argument "messages"", "proxy_server_request": {"url": "https://api.ai.it.ufl.edu/v1/embeddings", "method": "POST", "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "authorization": "Bearer REDACTED", "content-length": "75"}, "body": {"input": "Your text herhhe", "model": "sfr-embedding-mistral"}}, "user": "default_user_id", "metadata": {"user_api_key": "REDACTED", "user_api_key_metadata": {}, "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "content-length": "75"}, "user_api_key_alias": None, "user_api_key_user_id": "default_user_id", "user_api_key_team_id": None, "user_api_key_team_alias": None, "endpoint": "https://api.ai.it.ufl.edu/v1/embeddings", "model_group": "sfr-embedding-mistral", "deployment": "openai/Salesforce/SFR-Embedding-Mistral", "model_info": {"id": "b8c1399b-956b-8e54-b8fe-03c0d39ac4f6", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "api_base": "http://dgx-s23.ai.apps.it.ufl.edu:8007/", "caching_groups": None}, "model": "sfr-embedding-mistral", "input": "Your text herhhe"}, {"exception_type": "TypeError", "exception_string": "AsyncEmbeddings.create() got an unexpected keyword argument "messages"", "proxy_server_request": {"url": "https://api.ai.it.ufl.edu/v1/embeddings", "method": "POST", "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "authorization": "Bearer sk-6REDACTED", "content-length": "75"}, "body": {"input": "Your text herhhe", "model": "sfr-embedding-mistral"}}, "user": "default_user_id", "metadata": {"user_api_key": "REDACTED", "user_api_key_metadata": {}, "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "content-length": "75"}, "user_api_key_alias": None, "user_api_key_user_id": "default_user_id", "user_api_key_team_id": None, "user_api_key_team_alias": None, "endpoint": "https://api.ai.it.ufl.edu/v1/embeddings", "model_group": "sfr-embedding-mistral", "deployment": "openai/Salesforce/SFR-Embedding-Mistral", "model_info": {"id": "55f72312-eb20-7afb-e813-667a9d374e17", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "api_base": "http://dgx-s29.ai.apps.it.ufl.edu:8007/", "caching_groups": None}, "model": "sfr-embedding-mistral", "input": "Your text herhhe"}, {"exception_type": "ValueError", "exception_string": "No deployments available for selected model, passed model=sfr-embedding-mistral", "proxy_server_request": {"url": "https://api.ai.it.ufl.edu/v1/embeddings", "method": "POST", "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "authorization": "Bearer sk-REDACTED", "content-length": "75"}, "body": {"input": "Your text herhhe", "model": "sfr-embedding-mistral"}}, "user": "default_user_id", "metadata": {"user_api_key": "REDACTED", "user_api_key_metadata": {}, "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "content-length": "75"}, "user_api_key_alias": None, "user_api_key_user_id": "default_user_id", "user_api_key_team_id": None, "user_api_key_team_alias": None, "endpoint": "https://api.ai.it.ufl.edu/v1/embeddings", "model_group": "sfr-embedding-mistral", "deployment": "openai/Salesforce/SFR-Embedding-Mistral", "model_info": {"id": "55f72312-eb20-7afb-e813-667a9d374e17", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "api_base": "http://dgx-s29.ai.apps.it.ufl.edu:8007/", "caching_groups": None}, "model": "sfr-embedding-mistral", "input": "Your text herhhe"}, {"exception_type": "TypeError", "exception_string": "AsyncEmbeddings.create() got an unexpected keyword argument "messages"", "proxy_server_request": {"url": "https://api.ai.it.ufl.edu/v1/embeddings", "method": "POST", "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "authorization": "Bearer sk-REDACTED", "content-length": "75"}, "body": {"input": "Your text herhhe", "model": "sfr-embedding-mistral"}}, "user": "default_user_id", "metadata": {"user_api_key": "REDACTED", "user_api_key_metadata": {}, "headers": {"host": "api.ai.it.ufl.edu", "user-agent": "curl/8.1.2", "accept": "/", "content-type": "application/json", "content-length": "75"}, "user_api_key_alias": None, "user_api_key_user_id": "default_user_id", "user_api_key_team_id": None, "user_api_key_team_alias": None, "endpoint": "https://api.ai.it.ufl.edu/v1/embeddings", "model_group": "sfr-embedding-mistral", "deployment": "openai/Salesforce/SFR-Embedding-Mistral", "model_info": {"id": "b8c1399b-956b-8e54-b8fe-03c0d39ac4f6", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "api_base": "http://dgx-s23.ai.apps.it.ufl.edu:8007/", "caching_groups": None}, "model": "sfr-embedding-mistral", "input": "Your text herhhe"}]}, "model_info": {"id": "55f72312-eb20-7afb-e813-667a9d374e17", "max_tokens": 4096, "input_cost_per_token": 1e-09}, "timeout": None, "max_retries": 0, "litellm_call_id": "0a765bd1-5030-4926-89ba-ec1d55de3787", "litellm_logging_obj": <litellm.utils.Logging object at 0x7f14a8296b90>, "preset_cache_key": "402d851eb0e39502edf3481c6f5903591bf292a9b1bdb06690abaad9bd54e6b4"}

[ncecere]

Sorry about the comically long log output and for the late reply

[krrishdholakia]

@ncecere got it - the issue would be from the openai client which doesn't accept messages as a kwarg when making a request. We can fix this by passing it in as extra body

Just so i understand this - your salesforce model accepts a messages kwarg for embedding requests?

[ncecere]

when I'm adding this embedding model via curl to litellm I'm only passing in cost to the litellm_params. The messages section seems to get added automatically. which I think is causing the issue I'm having. it seems to be fixed in latest stable release though

[ncecere]

You can close this ticket, all good now