use tool config for gemini pro function calling

[krrishdholakia]

No description provided.

[krrishdholakia]

https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling#curl_2

tried using this locally with toolconfig. This seems like it would only work with later versions of the vertex ai sdk.

Considering we should just move to making http calls (like we do for anthropic models) - will allow everyone to use the latest gemini features without needing to upgrade litellm + vertex ai

[krrishdholakia]

picking this up today. got pushed back due to refactoring work over the weekend.

[guiramos]

@krrishdholakia thanks. Not sure if you know this, but Google Gemini is capable of automatically calling the function for you.

In the list of tools, you can pass an instance of CallableFunctionDeclaration, for example:

) long_term_memory_query_function = CallableFunctionDeclaration(
name="long_term_memory_query_function",
function=self.functions["long_term_memory_query_function"],
description="Query assistant long-term memory to remember and recall data. Indicated to retrieve personal and contextual information.",
parameters={
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The query string"
},
"document_id": {
"type": "string",
"description": "The id of the entry in the long-term memory. Only provide the document_id if you are sure about the variable name."
}
},
"required": ["query"]
}

In this example, self.functions["long_term_memory_query_function"] points to method with the same signature as described in the schema.

Then, when you start the chat, you can pass something like this:
model.start_chat(enable_automatic_function_calling=True,

Please let us know your thoughts

[krrishdholakia]

that looks pretty similar to how openai's function calling works right @guiramos with tool_choice="auto"

btw - this should already be live in v1.40.16.

https://docs.litellm.ai/docs/providers/gemini#tool-calling

rom litellm import completion
import os
# set env
os.environ["GEMINI_API_KEY"] = ".."

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]

response = completion(
    model="gemini/gemini-1.5-flash",
    messages=messages,
    tools=tools,
    tool_choice="auto" 
)

[guiramos]

@krrishdholakia well, not reallly.

Google gemini will actually make the call to your function, openai api won't do that. You have to parse the response, build the call to the function yourself.

[krrishdholakia]

that's interesting - can you point me to docs on this @guiramos i'll see how to add it

we can probably create a separate issue for tracking that

[guiramos]

@krrishdholakia
Even when I was researching myself I was not able to find the official documentation for this but I can tell you it works.
This is what I have:

 list_conversations_function = CallableFunctionDeclaration(
            name="list_conversations_function",
            function=self.functions["list_conversations_function"],
            description="List all conversations available with the user.",
            parameters={
                "type": "object",
                "properties": {}
            }
        )

 select_conversation_function = CallableFunctionDeclaration( name="select_conversation_function",
            function=self.functions["select_conversation_function"],
            description="Select the conversation by its title to be the active one.",
            parameters={
                "type": "object",
                "properties": {
                    "conversation_title": {
                        "type": "string",
                        "description": "The title of the conversation to select"
                    }
                },
                "required": ["conversation_title"]
            }
        )
        
        agent_tool = Tool(
            function_declarations=[
                list_conversations_function,
                select_conversation_function,
                
            ]
        )

        function_calling_config = FunctionCallingConfig()
        function_calling_config.mode = FunctionCallingConfig.Mode.ANY
        function_calling_config.allowed_function_names.extend([
            "list_conversations_function",
            "select_conversation_function"
        ])

Again, pay attention to the function attribute which needs to point to the actual method that will get called automatically.

You can use FunctionCallingConfig.Mode.AUTO too without providing the name of the functions.

self.model = genai.GenerativeModel(
            model_name=self.model_name,
            system_instruction=self.system_instruction,
            tools=self.agent_tool,
            tool_config=self.tool_config
        )
        self.chat = self.model.start_chat(enable_automatic_function_calling=True, history=chat_history)

The enable_automatic_function_calling=True is necessary for automatic function call.

This is my method I have the custom call for the chatcreated by gemini api, for some reason I had to pass the tools and tools_config again. Not only n the constructor for the model (above).

def send_message(self, message_content, json_mode=False, json_schema: Optional[List] = None, temperature=0.0):
        return self.chat.send_message(content=convert_message_to_gemini(message_content),
                                      generation_config=build_config(json_mode=json_mode, json_schema=json_schema, temperature=temperature),
                                      safety_settings=self.safety_settings,
                                      tools=self.agent_tool,
                                      tool_config=self.tool_config
                                      )

This send_message method I plan to replace with your litellm implementation.

[krrishdholakia]

what is self.functions?

[guiramos]

@krrishdholakia

In my implementations is an array of function points that matches the one I am passing to gemini.

functions = {
            "list_conversations_function": self.list_conversations_function_handler,
            "select_conversation_function": self.select_conversation_function_handler,
        }

        self.prepare_system_message()
        self.chat = GoogleChat(system_instruction=self.system_prompt,
                               functions=functions
                               )
                               

The self in the self.functions you are asking about is the GoogleChatinstance.

[guiramos]

@krrishdholakia

Did you give up?

[krrishdholakia]

hi @guiramos, dealing with some p0s.

Would welcome a PR for this.