Python: Bug: using Azure AI agents with tools from SK

[kenzabenjelloun]

Describe the bug
I've been trying to use Azure AI Agents with Semantic Kernel, it works fine when I add Semantic Kernel Plugins, but when I have tools defined directly in my AI Agent (that appears in AI foundry) beforehand, I get the following error :

Traceback (most recent call last):
  File ".venv\Lib\site-packages\semantic_kernel\kernel.py", line 357, in invoke_function_call
    function_to_call = self.get_function(function_call.plugin_name, function_call.function_name)
  File ".venv\Lib\site-packages\semantic_kernel\functions\kernel_function_extension.py", line 292, in get_function
    raise KernelFunctionNotFoundError(f"Function '{function_name}' not found in any plugin.")
semantic_kernel.exceptions.kernel_exceptions.KernelFunctionNotFoundError: Function 'my_function' not found in any plugin.

This is how i initialize my agent in SK so far :

   agent_definition = await _project_client.agents.get_agent(agent_id=os.getenv("FLIGHT_PARAMS_EXTRACTION_AGENT_ID"))

  
    agent = AzureAIAgent(
        client=_project_client,
        definition=agent_definition,
        kernel=kernel
    )

And this is how my AI agent is created:

functions_to_use: Set = {
    function, 
} 

# Create function definitions for the agent 
functions = FunctionTool(functions=functions_to_use)

    agent = project_client.agents.create_agent(
        model=model,
        name="Agent",
        instructions=instructions,
        response_format=response_format,
        },
        temperature=0.1,
        tools=functions.definitions
    )

Platform

• Language: Python
• SK version:1.29.0
• AI model: GPT 4o

[moonbox3]

Hi @kenzabenjelloun, all tool calling is handled via SK's kernel. If you previously add function.defintions to your agent's tools, then those same functions will need to be present for SK to leverage. This is the reason why you're seeing the error related to a tool not being found.

This intro example shows how to best use tools with the AzureAIAgent: the actual agent definition is pretty slim, and then add the plugins directly to the AzureAIAgent constructor (you can also provided a kernel instance with the plugins already added, if you wish):

semantic-kernel/python/samples/getting_started_with_agents/azure_ai_agent/step2_azure_ai_agent_plugin.py

Line 63 in fa91988

plugins=[MenuPlugin()], # Add the plugin to the agent

[moonbox3]

@kenzabenjelloun as part of SK Python 1.31.0, that we just released, there is new validation that will make sure that any function tools registered on the agent, also must be present in the kernel.