.Net Function Calling with Bedrock Claude

[MalteFries]

Hey there, am I missing something here or does function calling for Claude 3.5 Sonnet via the Bedrock connector simply not work?
I know it's still in experimental, but is there a plan to support it?

[Douuuglas]

Hey @MalteFries, unfortunately it's not working at the moment.

If you look at the code below it's expecting us to pass "tools" and "tool_choice" objects in the ExtensionData.

https://github.com/microsoft/semantic-kernel/blame/7a61c9caff11f73bed93687144de3afda85fd39d/dotnet/src/Connectors/Connectors.Amazon/Bedrock/Core/Models/Anthropic/AnthropicService.cs#L71C9-L82C10

But ClaudeToolUse.ClaudeTool is internal and it's not available outside the SK lib, so we can't use it.

@markwallace-microsoft, is anyone working on it? If I have some guidance I could try creating a PR.

My idea would be doing something similar as what happens on the OpenAI Connector, removing the need to check the ExtesionData.

semantic-kernel/dotnet/src/Connectors/Connectors.OpenAI/Core/ClientCore.ChatCompletion.cs

Line 150 in 7a61c9c

var functionCallingConfig = this.GetFunctionCallingConfiguration(kernel, chatExecutionSettings, chatHistory, requestIndex);

[halllo]

I also need function calling with bedrock (claude).

In my experiments I get an the following exception as soon as I add PromptExecutionSettings with FunctionChoiceBehavior.Auto():

Microsoft.SemanticKernel.Connectors.Amazon.Core.BedrockChatCompletionClient[0]
      Can't converse with 'anthropic.claude-3-sonnet-20240229-v1:0'. Reason: 1 validation error detected: Value '0' at 'inferenceConfig.maxTokens' failed to satisfy constraint: Member must have value greater than or equal to 1
      Amazon.BedrockRuntime.Model.ValidationException: 1 validation error detected: Value '0' at 'inferenceConfig.maxTokens' failed to satisfy constraint: Member must have value greater than or equal to 1

If it is really just a matter of an inaccessible property, why not get it using reflection? The connector is experimental after all.

[Douuuglas]

I also need function calling with bedrock (claude).

In my experiments I get an the following exception as soon as I add PromptExecutionSettings with FunctionChoiceBehavior.Auto():

Microsoft.SemanticKernel.Connectors.Amazon.Core.BedrockChatCompletionClient[0]
      Can't converse with 'anthropic.claude-3-sonnet-20240229-v1:0'. Reason: 1 validation error detected: Value '0' at 'inferenceConfig.maxTokens' failed to satisfy constraint: Member must have value greater than or equal to 1
      Amazon.BedrockRuntime.Model.ValidationException: 1 validation error detected: Value '0' at 'inferenceConfig.maxTokens' failed to satisfy constraint: Member must have value greater than or equal to 1

If it is really just a matter of an inaccessible property, why not get it using reflection? The connector is experimental after all.

Hey @halllo,

To make it work I had to set the "max_tokens_to_sample" property manually as you can see below:

PromptExecutionSettings promptExecutionSettings = new()
{
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
    ModelId = modelid,
    ExtensionData = new Dictionary<string, object>() {
        { "max_tokens_to_sample", 4096 }
    },
};

And then I pass the prompt execution settings while calling the chat completion service:

var result = await chatcompletionservice.GetChatMessageContentAsync(
    history,
    executionSettings: promptExecutionSettings,
    kernel: kernel);

See if that works for you!

[halllo]

Hello @Douuuglas.

Thanks for taking the time to take a look. With your proposed change I don’t see the exception anymore, but it also does not invoke the function. Is it invoking the function on your end?

[Douuuglas]

Nice @halllo,

Unfortunately it won't work as I stated in the comment below, still needs to be implemented.

#9750 (comment)

[halllo]

I have now built my own function calling abstraction over the IAmazonBedrockRuntime:

await agent.Do(
   task: "Get the most popular song played on the radio station RGBG and rate it as bad.",
   tools:
   [
      Agent.Tool.From([Description("Get radio song")]([Description("The call sign for the radio station for which you want the most popular song."), Required] string sign)
      => new { songName = "Random Song 1" }),

      Agent.Tool.From([Description("Rate a song")](string song, string rating)
      => "Rated!"),
   ]);

This is the tool specification serialization: https://github.com/halllo/DotnetAgentExperiments/blob/main/aws.bed/Agent.cs#L123

[brused27]

How do we get some traction on this issue?

I stumbled across this myself and I thought I was doing something wrong since my plugins are correctly auto-invoking against the Azure OpenAI connector but not on the Bedrock Claude connector. It would be nice if the library itself would throw an exception or perhaps log a warning that we can't use plugins on this connector if it is truly unsupported today - it would save developers time on digging for a resolution.