gemini (vertex): add tool function calling support

[semanser]

Hi, is there any specific reason why function calling wasn't implemented for Vertex in #709? I may give it a try, but I just want to make sure that there are no known blockers.

langchaingo/llms/googleai/googleai.go

Lines 180 to 183 in 33b8795

	case schema.ChatMessageTypeFunction, schema.ChatMessageTypeTool:
	fallthrough
	default:
	return nil, fmt.Errorf("role %v not supported", content.Role)

Resources:

• https://pkg.go.dev/cloud.google.com/go/vertexai/genai#Tool
• https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#overview

[semanser]

I started implementing this but immediately found an issue: function calling is supported in Vertex (cloud.google.com/go/vertexai/genai) but not in github.com/google/generative-ai-go/genai.

vertex.go is generated based on the googleai.go with the main difference that GenerativeModel returns different models in both files. That means that we can't use vertexGenAI "cloud.google.com/go/vertexai/genai" inside of googleai.go...

[semanser]

In other words, we can't add any vertex-specific logic inside of googleai.go, and the only way to do that is to change it via generate-vertex.go. This might increase the difference between the two files (vertex and googleai), but I'm not sure if it's significant enough to stopp the use of the generator. I will wait for input from @tmc here.

[eliben]

Tool calling is implemented in the latest version of googleai -- see https://pkg.go.dev/github.com/google/generative-ai-go@v0.10.0/genai#example-Tool

Happy to review PRs here

[eliben]

@semanser are you planning to continue working on this in the immediate future? I want this implemented, so I was thinking of tackling it myself unless you have a PR in progress already

[semanser]

@eliben I'm not planning to, so feel free to continue 👍

[eliben]

The Google AI part (ai.google.dev) is done in #794
I'll look into implementing this for the Vertex side as well

[eliben]

#795 adds vertex ai support.

Next, an example and this issue can be closed