[JS] VertexAI.ClientError when providing context from retriever and Gemini calling a provided tool

[jeff-treuting]

Describe the bug
When calling generate with an array of available tools and providing the context parameter the results of a Firestore Retriever call, the call errors out with the following error:

{"severity":"ERROR","message":"Unhandled error Error: Vertex response generation failed: ClientError: [VertexAI.ClientError]: Within a single message, FunctionResponse cannot be mixed with other type of part in the request for sending chat message.\n    at /xxxx/node_modules/@genkit-ai/vertexai/lib/gemini.js:514:17\n    at Generator.throw (<anonymous>)\n    at rejected (/xxxx/node_modules/@genkit-ai/vertexai/lib/gemini.js:50:29)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"}

To Reproduce
Here is some simplified code that shows the setup that produces the error for me every time. When I comment out passing in the context, it works as expected.

const myTool = defineTool(
  {
    name: "myTool",
    description:
      "When a question about the calendar bookings or reservations is asked, this tool will load the calendar booking data.",
    inputSchema: z.object({
      beginDate: z
        .string()
        .optional()
        .describe("The beginning date to load the calendar data for."),
      endDate: z
        .string()
        .optional()
        .describe("The end date to load the calendar data for."),
      member: z
        .string()
        .optional()
        .describe("The member ID or name to filter the calendar bookings by."),
    }),
    outputSchema: z.string(),
  },
  async (input) => "haha Just kidding no joke about for you! got you"
);

    const myRetrieverRef = defineFirestoreRetriever({
      name: "retriever-id",
      firestore: getFirestore(),
      collection: "calendar/vectorIndex",
      contentField: "content",
      vectorField: "embedding",
      embedder: textEmbeddingGecko,
      distanceMeasure: "COSINE", // 'EUCLIDEAN', 'DOT_PRODUCT', or 'COSINE' (default)
    });

    const docs = await retrieve({
      retriever: myRetrieverRef,
      query: userMessage,
      options: {
        limit: 5,
        k: 3,
      },
    });

    const myPrompt = await prompt("myBot");
    const result = await myPrompt.generate({
      input: {
        message: userMessage,
        today: format(new Date(), "yyyy-MM-dd"), // "2022-01-01
        user: {
          id: request.auth?.uid || "anonymous",
          name: request.auth?.token.name || "anonymous",
        },
      },
      context: docs, // when I comment out this line the error goes away regardless of what tool I include, I've tried a few different versions of it
      tools: [myTool],
    });
    ```

**Expected behavior**
I would expect to be able to get results that utilize a cusotm tool for retrieving information form a data source and provide context from a retriever as well.


**Runtime (please complete the following information):**
 - OS: MacOS
 - Version: Sonoma 14.4.1
** Node version
 -  v20.11.1

[ceit-each]

I am hitting the same issue.

[jeff-treuting]

It appears the issue is form Gemini. Here is the place in the gemini code where the error message comes from and seems like it is not supported at this point:

https://github.com/google-gemini/generative-ai-js/blob/5739e2a32736a2c8b1e2689e0d68dd3565b3baae/packages/main/src/requests/request-helpers.ts#L95

[cabljac]

Hi, just to clarify, is this using the Vertex AI plugin? The package you linked is from Google AI, but the title of the issue mentions Vertex.

Could you provide your configureGenkit options? (redacting anything sensitive of course)

[jeff-treuting]

import { firebase } from "@genkit-ai/firebase";
import { vertexAI } from "@genkit-ai/vertexai";
import { dotprompt } from "@genkit-ai/dotprompt";

configureGenkit({
  plugins: [firebase(), vertexAI({ location: "us-central1" }), dotprompt()],
  logLevel: "debug",
  traceStore: "firebase",
  enableTracingAndMetrics: true,
});

But the model in my dotprompt file refers to gemini-1.5-flash-preview (and I've tried with gemini-1.0-pro as well).

model: vertexai/gemini-1.5-flash-preview

[cabljac]

Just noting down what i've found so far:

Seems like the issue is in this method interacting with the Vertex models, in that a function response part can't also have parts of other type in it.

Note that all the Vertex AI plugin models do not contain context under info.supports, so they will be using this middleware.

[cabljac]

OK so appears to be exactly what it's logging actually, a FunctionResponse is being returned to Gemini together with other types and it doesn't like that.

My theory is that the ToolResponse contains the context still, and this will be added as text parts to the gemini request, causing it to complain.

In fact this line https://github.com/firebase/genkit/blob/main/js/plugins/vertexai/src/gemini.ts#L515 is likely the issue, as we send all ToolResponses and other type parts back together.

[cabljac]

I believe this is now fixed, as of #591