[Bug]: Litellm does not work with AWS Bedrock Application Inference profiles

[mirodrr2]

What happened?

To reproduce:

1. Create application inference profile:

aws bedrock create-inference-profile \
  --inference-profile-name "Claude37" \      
  --description "Application profile for Claude 3.7 Sonnet" \
  --model-source copyFrom=arn:aws:bedrock:us-east-1:<Your-Account-Id>:inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0 \
  --tags key=projectId,value=ai-gateway
{
    "inferenceProfileArn": "<Your application inference profile arn>",
    "status": "ACTIVE"
}

2. Configure the inference profile arn as a model in your config.yaml. This matches the same way you would use it with the boto3 client

- model_name: anthropic.claude-3-7-sonnet-20250219-v1:0-profile
    litellm_params:
      model: bedrock/converse/<Your application inference profile arn>

3. Call the model_name linked to your application inference profile

curl -i "$GATEWAY_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GATEWAY_API_KEY" \
-d '{
    "model": "anthropic.claude-3-7-sonnet-20250219-v1:0-profile",
    "messages": [
        {
            "role": "user",
            "content": "tell me a one sentence story"
        }
    ]
}'

HTTP/2 200 
date: Sat, 01 Mar 2025 02:55:22 GMT
content-type: application/json
content-length: 1106
server: uvicorn
server: uvicorn
x-litellm-version: 1.61.20
x-litellm-response-cost: 0
x-litellm-key-spend: 0.0
x-litellm-timeout: 6000

{"error": {"message": "litellm.APIConnectionError: 'output'\nTraceback (most recent call last):\n  File \"/usr/lib/python3.13/site-packages/litellm/main.py\", line 470, in acompletion\n    response = await init_response\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.13/site-packages/litellm/llms/bedrock/chat/converse_handler.py\", line 243, in async_completion\n    return litellm.AmazonConverseConfig()._transform_response(\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n        model=model,\n        ^^^^^^^^^^^^\n    ...<8 lines>...\n        encoding=encoding,\n        ^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/usr/lib/python3.13/site-packages/litellm/llms/bedrock/chat/converse_transformation.py\", line 656, in _transform_response\n    message: Optional[MessageBlock] = completion_response[\"output\"][\"message\"]\n                                      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^\nKeyError: 'output'\n. Received Model Group=anthropic.claude-3-7-sonnet-20250219-v1:0-profile\nAvailable Model Group Fallbacks=None", "type": null, "param": null, "code": "500"}}%     

This same request, but using the boto3 client against bedrock works just fine:

Example script that runs fine:

import boto3
import os
from botocore.client import Config
from botocore import UNSIGNED
from typing import Generator, Dict, Any, Optional


def create_bedrock_client():
    
    region = os.getenv("AWS_REGION")

    if not all([region]):
        raise ValueError(
            "Missing required environment variables: AWS_REGION"
        )
    
    session = boto3.Session()
    client = session.client(
        "bedrock-runtime",
        region_name=region,
    )
    return client

bedrock_client = create_bedrock_client()
messages = [{"role": "user", "content": [{"text": "Create a list of 3 pop songs."}]}]
model_id = "arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk"
response = bedrock_client.converse(modelId=model_id, messages=messages)
print(response)

Output:

{'ResponseMetadata': {'RequestId': 'e684f90b-a4af-4d5c-805c-590160849f4a', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Fri, 28 Feb 2025 20:14:11 GMT', 'content-type': 'application/json', 'content-length': '435', 'connection': 'keep-alive', 'x-amzn-requestid': 'e684f90b-a4af-4d5c-805c-590160849f4a'}, 'RetryAttempts': 0}, 'output': {'message': {'role': 'assistant', 'content': [{'text': '# 3 Popular Songs\n\n1. "Billie Jean" by Michael Jackson\n2. "Shape of You" by Ed Sheeran\n3. "Uptown Funk" by Mark Ronson ft. Bruno Mars'}]}}, 'stopReason': 'end_turn', 'usage': {'inputTokens': 17, 'outputTokens': 54, 'totalTokens': 71}, 'metrics': {'latencyMs': 6475}}

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.61.20.rc

Twitter / LinkedIn details

No response

[mirodrr2]

I also added some new logs in the converse_handler.async_completion to get more insight.

NEW_LOG: model arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk |  

NEW_LOG: request_data {'messages': [{'role': 'user', 'content': [{'text': 'tell me a one sentence story'}]}], 'additionalModelRequestFields': {}, 'system': [], 'inferenceConfig': {}}

NEW_LOG: data" {"messages": [{"role": "user", "content": [{"text": "tell me a one sentence story"}]}], "additionalModelRequestFields": {}, "system": [], "inferenceConfig": {}} api_base: https://bedrock-runtime.us-east-1.amazonaws.com/model/arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk/converse headers: https://bedrock-runtime.us-east-1.amazonaws.com/model/arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk/converse

NEW_LOG: resonse" <Response [200 OK]>

NEW_LOG: completion_response: {'Output': {'__type': 'com.amazon.coral.service#UnknownOperationException'}, 'Version': '1.0'}

This is the file and method I added logs to. Modified version is below:

litellm/litellm/llms/bedrock/chat/converse_handler.py

Line 165 in c8dc4f3

async def async_completion(

async def async_completion(
        self,
        model: str,
        messages: list,
        api_base: str,
        model_response: ModelResponse,
        timeout: Optional[Union[float, httpx.Timeout]],
        encoding,
        logging_obj: LiteLLMLoggingObject,
        stream,
        optional_params: dict,
        litellm_params: dict,
        credentials: Credentials,
        logger_fn=None,
        headers: dict = {},
        client: Optional[AsyncHTTPHandler] = None,
    ) -> Union[ModelResponse, CustomStreamWrapper]:
        print(f'NEW_LOG: model {model}')

        request_data = await litellm.AmazonConverseConfig()._async_transform_request(
            model=model,
            messages=messages,
            optional_params=optional_params,
            litellm_params=litellm_params,
        )
        print(f'NEW_LOG: request_data {request_data}')
        data = json.dumps(request_data)

        prepped = self.get_request_headers(
            credentials=credentials,
            aws_region_name=litellm_params.get("aws_region_name") or "us-west-2",
            extra_headers=headers,
            endpoint_url=api_base,
            data=data,
            headers=headers,
        )

        ## LOGGING
        logging_obj.pre_call(
            input=messages,
            api_key="",
            additional_args={
                "complete_input_dict": data,
                "api_base": api_base,
                "headers": prepped.headers,
            },
        )

        headers = dict(prepped.headers)
        if client is None or not isinstance(client, AsyncHTTPHandler):
            _params = {}
            if timeout is not None:
                if isinstance(timeout, float) or isinstance(timeout, int):
                    timeout = httpx.Timeout(timeout)
                _params["timeout"] = timeout
            client = get_async_httpx_client(
                params=_params, llm_provider=litellm.LlmProviders.BEDROCK
            )
        else:
            client = client  # type: ignore

        try:
            print(f'NEW_LOG: data" {data} api_base: {api_base} headers: {api_base}')
            response = await client.post(
                url=api_base,
                headers=headers,
                data=data,
                logging_obj=logging_obj,
            )  # type: ignore
            response.raise_for_status()
            print(f'NEW_LOG: resonse" {response}')
        except httpx.HTTPStatusError as err:
            print(f'NEW_LOG: err" {err}')
            error_code = err.response.status_code
            raise BedrockError(status_code=error_code, message=err.response.text)
        except httpx.TimeoutException:
            raise BedrockError(status_code=408, message="Timeout error occurred.")

        return litellm.AmazonConverseConfig()._transform_response(
            model=model,
            response=response,
            model_response=model_response,
            stream=stream if isinstance(stream, bool) else False,
            logging_obj=logging_obj,
            api_key="",
            data=data,
            messages=messages,
            optional_params=optional_params,
            encoding=encoding,
        )

[mirodrr2]

I have a feeling maybe the boto3 client is detecting that an arn has been passed in and is doing some pre-processing before making the api call? Not sure, would need to look at the boto3 source code or do wireshark to see the final network call

[mirodrr2]

Application inference profiles if you're unfamiliar with them: https://community.aws/content/2oa6QAE7tgxmAsgAExXY2B2MqCV/getting-started-with-bedrock-application-inference-profile?lang=en

{"error": {"message": "litellm.APIConnectionError: 'output'\nTraceback (most recent call last):\n File "/usr/lib/python3.13/site-packages/litellm/main.py", line 470, in acompletion\n response = await init_response\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/lib/python3.13/site-packages/litellm/llms/bedrock/chat/converse_handler.py", line 243, in async_completion\n return litellm.AmazonConverseConfig()._transform_response(\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n model=model,\n ^^^^^^^^^^^^\n ...<8 lines>...\n encoding=encoding,\n ^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/usr/lib/python3.13/site-packages/litellm/llms/bedrock/chat/converse_transformation.py", line 656, in _transform_response\n message: Optional[MessageBlock] = completion_response["output"]["message"]\n ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^\nKeyError: 'output'\n. Received Model Group=anthropic.claude-3-7-sonnet-20250219-v1:0-profile\nAvailable Model Group Fallbacks=None", "type": null, "param": null, "code": "500"}}%

Hey @mirodrr2 it looks like the response is not in the standard converse format

Can you share the debug logs, specifically the raw response that is received by litellm

[mirodrr2]

I did share the debug logs, in this comment: #8911 (comment)

This is the most relevant one:

NEW_LOG: completion_response: {'Output': {'__type': 'com.amazon.coral.service#UnknownOperationException'}, 'Version': '1.0'}

It's throwing an error. So the request is invalid, not the response. It seems boto3 is doing something special with the arn that litellm is not doing

Hey @mirodrr2 what's the raw request made by litellm?

[mirodrr2]

@krrishdholakia

you can see the boto3 request by routing it via the proxy by specifying the endpoint_url

https://docs.litellm.ai/docs/pass_through/bedrock#advanced---bedrock-agents

then run the proxy with --detailed_debug and see the raw request received by litellm in the logs

[mirodrr2]

This is the most relevant log to answer that question, also from that same comment

NEW_LOG: data" {"messages": [{"role": "user", "content": [{"text": "tell me a one sentence story"}]}], "additionalModelRequestFields": {}, "system": [], "inferenceConfig": {}}

api_base: https://bedrock-runtime.us-east-1.amazonaws.com/model/arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk/converse

headers: https://bedrock-runtime.us-east-1.amazonaws.com/model/arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk/converse

(Disregard headers, bug in my log)

missed the api base, thanks

[mirodrr2]

Okay, away from computer right now, but will do that boto3 request through the gateway when I get a chance. Good idea, thanks

[mirodrr2]

Okay, I did it, and it turns out the boto3 client url encodes the :s in the arn when it's used as a model id.

"POST /bedrock/model/arn%3Aaws%3Abedrock%3Aus-east-1%3A235614385815%3Aapplication-inference-profile/cmop7wjzk1jk/converse

I assume just url encoding all bedrock model ids would work fine as a fix?

@krrishdholakia

Or actually, that might not be it, as other model ids already have colons...

[mirodrr2]

Also that method of calling the gateway through the bedrock passthrough api doesn't seem to work? This is my script (based off the script from your linked docs)

import boto3
import os
from botocore.client import Config
from botocore import UNSIGNED
from typing import Generator, Dict, Any, Optional

proxy_endpoint = "https://litellm.mirodrr.people.aws.dev/bedrock"

custom_headers = {
    'litellm_user_api_key': 'sk-1234', # 👈 your proxy api key
}


def create_bedrock_client():
    
    region = os.getenv("AWS_REGION")

    if not all([region]):
        raise ValueError(
            "Missing required environment variables: AWS_REGION"
        )
    
    session = boto3.Session()
    client = session.client(
        "bedrock-runtime",
        region_name=region,
        endpoint_url=proxy_endpoint
    )
    return client
def inject_custom_headers(request, **kwargs):
    request.headers.update(custom_headers)

bedrock_client = create_bedrock_client()

# Attach the event to inject custom headers before the request is sent
bedrock_client.meta.events.register('before-send.*.*', inject_custom_headers)

messages = [{"role": "user", "content": [{"text": "Create a list of 3 pop songs."}]}]
model_id = "arn:aws:bedrock:us-east-1:235614385815:application-inference-profile/cmop7wjzk1jk"
response = bedrock_client.converse(modelId=model_id, messages=messages)
print(response)

And I just get this error:

fresh_venv[ mirodrr | ~/scripts] > python3 boto3-raw.py
Traceback (most recent call last):
  File "/Users/mirodrr/scripts/boto3-raw.py", line 40, in <module>
    response = bedrock_client.converse(modelId=model_id, messages=messages)
  File "/Users/mirodrr/scripts/fresh_venv/lib/python3.13/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mirodrr/scripts/fresh_venv/lib/python3.13/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (401) when calling the Converse operation: 

And these logs:

�[92m05:02:15 - LiteLLM Proxy:ERROR�[0m: user_api_key_auth.py:1087 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Malformed API Key passed in. Ensure Key has `Bearer ` prefix. Passed in: AWS4-HMAC-SHA256 Credential=ASIATNW5WG2LYM3Y4WCH/20250301/us-east-1/bedrock/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=1ba8e55fc7c15c2c6ca0f0b3a9bc7da7cb770596b848f905a1abd2eafa2a96c7
--
  | 2025-03-01T05:02:15.169Z | Requester IP Address:10.0.44.215
  | 2025-03-01T05:02:15.169Z | Traceback (most recent call last):
  | 2025-03-01T05:02:15.169Z | File "/usr/lib/python3.13/site-packages/litellm/proxy/auth/user_api_key_auth.py", line 540, in _user_api_key_auth_builder
  | 2025-03-01T05:02:15.169Z | raise Exception(
  | 2025-03-01T05:02:15.169Z | f"Malformed API Key passed in. Ensure Key has `Bearer ` prefix. Passed in: {passed_in_key}"
  | 2025-03-01T05:02:15.169Z | )
  | 2025-03-01T05:02:15.169Z | Exception: Malformed API Key passed in. Ensure Key has `Bearer ` prefix. Passed in: AWS4-HMAC-SHA256 Credential=ASIATNW5WG2LYM3Y4WCH/20250301/us-east-1/bedrock/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=1ba8e55fc7c15c2c6ca0f0b3a9bc7da7cb770596b848f905a1abd2eafa2a96c7
  | 2025-03-01T05:02:15.171Z | INFO: 10.0.44.215:4646 - "POST /bedrock/model/arn%3Aaws%3Abedrock%3Aus-east-1%3A235614385815%3Aapplication-inference-profile/cmop7wjzk1jk/converse HTTP/1.1" 401 Unauthorized

[mknet3]

Close related -> #8905