genai: design for automatic function calling #75
Conversation
This PR is a suggested design for automatic function calling in Go. There is no implementation yet. - Add a Function field to FunctionDeclaration. Populating this field enables the function to be automatically called. If you set the field manually, you must set schema manually as well. - Add NewCallableFunctionDeclaration. This takes a function and both infers its schema and populates the entire FunctionDefinition. This design supports three use cases: 1. Manual function calling: set schema but not function. 2. Auto function calling, user schema: set schema and function. 3. Auto function calling, inferred schema: call NewCallableFunctionDeclaration But it doesn't support: 4. Manual function calling, inferred schema except in the roundabout way of using NewCallableFunctionDeclaration and then setting the Function field to nil.
There was a problem hiding this comment.
Thanks for the design! It'd be nice to start the review with some sort of high-level documentation/example explaining how users use this. We'd have to include this documentation somewhere anyway, and it should probably just be in the docstrings so it renders to pkg.go.dev
| panic(err) | ||
| } | ||
|
|
||
| // Use the FunctionDeclaration to populate Model.Tools. |
There was a problem hiding this comment.
The example should probably be extended to show this?
There was a problem hiding this comment.
I started to do that. I can finish when I have an implementation.
genai/functions.go
Outdated
| "github.com/google/generative-ai-go/internal/support" | ||
| ) | ||
|
|
||
| // A Tool is a piece of code that enables the system to interact with |
There was a problem hiding this comment.
It's Tool in file functions.go but talking about functions - do we just use the two interchangeably? Aligning with how the Python SDK names things?
There was a problem hiding this comment.
Both OpenAI and Gemini group functions into "tools." The Python SDK follows that, and we do too.
There was a problem hiding this comment.
Right, so maybe we should structure it like that too - tool is the high level concept, and functions are one of the tools available. So for example the Tool type would be in tools.go, not functions.go?
Also, is containment the right paradigm here? Is a function a kind of tool, or do tools contain functions and potentially other stuff?
There was a problem hiding this comment.
Renamed the file.
In the protos, a Tool contains FunctionDeclarations. Python keeps that structure.
genai/functions.go
Outdated
| } | ||
| } | ||
|
|
||
| // NewCallableFunctionDeclaration creates a [FunctionDeclaration] from a Go |
There was a problem hiding this comment.
What's the significance of "Callable" here? Are there also non-callable function declarations?
There was a problem hiding this comment.
I mean "automatically callable." As in, the client will invoke the function for you, instead of returning a FunctionCall Part and having you provide the result. (The automatic calling isn't implemented yet.)
There was a problem hiding this comment.
So maybe removing the "Callable" can help make this a bit shorter without loss of meaning? After all, the type is FunctionDeclaration, so NewFunctionDeclaration for the constructor sounds logical
There was a problem hiding this comment.
After all, the type is FunctionDeclaration, so NewFunctionDeclaration for the constructor sounds logical
True, but usually a function named NewFoo does about the same thing as &Foo{...}. Here the New function also makes the function automatically callable.
|
I added a section to the package doc. |
This PR is a suggested design for automatic function calling in Go.
There is no implementation yet.
Add a Function field to FunctionDeclaration. Populating this
field enables the function to be automatically called. If you
set the field manually, you must set schema manually as well.
Add NewCallableFunctionDeclaration. This takes a function and
both infers its schema and populates the entire FunctionDefinition.
This design supports three use cases:
But it doesn't support:
except in the roundabout way of using NewCallableFunctionDeclaration
and then setting the Function field to nil.