[Issue]: AWS Bedrock Function calling

[SumaiyaSultan2002]

Describe the issue

I am working on autogen using meta.llama3-8b-instruct-v1:0 from AWS bedrock. And i face an issue where the agents are not using the function. Whereas while working with groq, it works well with function calling. @marklysze Can you please give me some ideas to fix this issue?

Steps to reproduce

No response

Screenshots and logs

No response

Additional Information

No response

[marklysze]

Hey @SumaiyaSultan2002, we should be merging in an Amazon Bedrock client soon, and that would be suitable to use. See #3232.

You could try out using that code if you wanted to.

Can you provide an example of the config settings (be mindful of removing keys and tokens) so we can see which client class it is using.

[SumaiyaSultan2002]

yes i have cloned and made use of the code. I find that it is not making use of the function for the agents.
"aws_region_name": "us-east-1",
"api_type": "bedrock",
"model": "meta.llama3-8b-instruct-v1:0",
"aws_access_key": " ",
"aws_secret_key": " ",
"price" : [0.003, 0.015],

[marklysze]

Thanks for that @SumaiyaSultan2002, just a note that the region parameter is aws_region not aws_region_name - there was an error in the comment for the Bedrock client class, apologies.

I tried the config and Bedrock returned:
An error occurred (ValidationException) when calling the Converse operation: This model doesn't support tool use.

It's a subtle model difference - Llama 3 doesn't support tools but Llama 3.1 does. Llama 3.1 is available in the us-west-2 AWS region. Model capabilities are here.

I've also found that the 8B model isn't as good at calling functions as the 70B model.

If you want to try Llama 3.1 70B in us-west-2 (the 3.1 70B model is cheaper than Llama 3 8B :) ):

        "api_type": "bedrock",
        "model": "meta.llama3-1-70b-instruct-v1:0",
        "aws_region": "us-west-2",
        "price": [0.00099, 0.00099],
        "cache_seed": None,  # turn off caching, optional
        "aws_access_key": "",
        "aws_secret_key": "",

I've also noticed that Llama 3.1 tends to pass numeric parameters as strings, in this demo code we have checked if the number parameter is a string and converted it to a float. This example is in the PR's documentation change.

import json
from typing import Literal, Annotated

import autogen

config_list_bedrock = [
    {
        "api_type": "bedrock",
        "model": "meta.llama3-1-70b-instruct-v1:0",
        "aws_region": "us-west-2",
        "price": [0.00099, 0.00099],
        "cache_seed": None,  # turn off caching, optional
        "aws_access_key": "",
        "aws_secret_key": "",
    }
]

# Create the agent and include examples of the function calling JSON in the prompt
# to help guide the model
chatbot = autogen.AssistantAgent(
    name="chatbot",
    system_message="""For currency exchange and weather forecasting tasks,
        only use the functions you have been provided with.
        Output only the word 'TERMINATE' when an answer has been provided.
        Use both tools together if you can.""",
    llm_config={
        "config_list": config_list_bedrock,
    },
)

user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    is_termination_msg=lambda x: x.get("content", "") and "TERMINATE" in x.get("content", ""),
    human_input_mode="NEVER",
    max_consecutive_auto_reply=2,
)

# Currency Exchange function

CurrencySymbol = Literal["USD", "EUR"]

# Define our function that we expect to call
def exchange_rate(base_currency: CurrencySymbol, quote_currency: CurrencySymbol) -> float:
    if base_currency == quote_currency:
        return 1.0
    elif base_currency == "USD" and quote_currency == "EUR":
        return 1 / 1.1
    elif base_currency == "EUR" and quote_currency == "USD":
        return 1.1
    else:
        raise ValueError(f"Unknown currencies {base_currency}, {quote_currency}")


# Register the function with the agent
@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Currency exchange calculator.")
def currency_calculator(
    base_amount: Annotated[float, "Amount of currency in base_currency, float values (no strings), e.g. 987.82"],
    base_currency: Annotated[CurrencySymbol, "Base currency"] = "USD",
    quote_currency: Annotated[CurrencySymbol, "Quote currency"] = "EUR",
) -> str:
    # If the amount is passed in as a string, e.g. "123.45", attempt to convert to a float
    if isinstance(base_amount, str):
        base_amount = float(base_amount)

    quote_amount = exchange_rate(base_currency, quote_currency) * base_amount
    return f"{format(quote_amount, '.2f')} {quote_currency}"


# Weather function
# Example function to make available to model
def get_current_weather(location, unit="fahrenheit"):
    """Get the weather for some location"""
    if "chicago" in location.lower():
        return json.dumps({"location": "Chicago", "temperature": "13", "unit": unit})
    elif "san francisco" in location.lower():
        return json.dumps({"location": "San Francisco", "temperature": "55", "unit": unit})
    elif "new york" in location.lower():
        return json.dumps({"location": "New York", "temperature": "11", "unit": unit})
    else:
        return json.dumps({"location": location, "temperature": "unknown"})


# Register the function with the agent
@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Weather forecast for US cities.")
def weather_forecast(
    location: Annotated[str, "City name"],
) -> str:
    weather_details = get_current_weather(location=location)
    weather = json.loads(weather_details)
    return f"{weather['location']} will be {weather['temperature']} degrees {weather['unit']}"

# start the conversation
res = user_proxy.initiate_chat(
    chatbot,
    message="What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday?",
    summary_method="reflection_with_llm",
)

print(res.summary["content"])

Let me know how you go with that.

[SumaiyaSultan2002]

Thank you so much for the detailed explanation

[marklysze]

Thank you so much for the detailed explanation

No problem, if you do test it can you let us know how you went and we'll assist further or close the issue if all is okay.

[SumaiyaSultan2002]

Closing the issue. As it was rectified.