Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions scripts/mcp-tests/gemini-smoke-test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

if (!process.env.GEMINI_API_KEY) {
console.error("Must set GEMINI_API_KEY to run this smoke test.");
process.exit(1);
}

const client = new Client({
name: "firebase-mcp-smoke-tester",
version: "0.0.1",
});

await client.connect(
new StdioClientTransport({
command: "../../lib/bin/firebase.js",
args: [
"experimental:mcp",
"--only",
"firestore,dataconnect,messaging,remoteconfig,crashlytics,auth,storage",
],
}),
);

const { tools } = await client.listTools();

const geminiTools = tools.map((tool) => {
return {
name: tool.name,
description: tool.description,
parameters: tool.inputSchema,
};
});

const response = await fetch(
`https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=${process.env.GEMINI_API_KEY}`,
{
method: "POST",
headers: {
"content-type": "application/json",
},
body: JSON.stringify({
toolConfig: {
functionCallingConfig: { mode: "auto" },
},
tools: [{ functionDeclarations: geminiTools }],
contents: [
{
parts: [{ text: "Call the firebase_list_apps tool." }],
},
],
}),
},
);

if (response.status === 200) {
console.dir(await response.json(), { depth: null });

console.log("✅ Passed smoke test!");
process.exit(0);
} else {
const rtext = await response.text();
try {
console.dir(JSON.parse(rtext), { depth: null });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we know if the response conforms to what the tool expect?

Do we need to do some sort of zod validation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of the smoke test is to make sure the Gemini API doesn't throw errors based on any of the tool input schemas provided - it's not testing actual tool calling. This is more of a sanity check than a complete solution, but is better than nothing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, what kind of error it may throw?

Does the Gemini API do any validation to make sure the response match the spec?

We can probably add that validation here later.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the tool schemas don't conform to the limited schema supported by Gemini, the request will 400 even if it's only one tool out of 35 that are broken. That's bad, so this smoke test ensures that all of our tools are vaild as far as the API is concerned.

} catch (e) {
console.log(rtext);
}
console.error("ERROR: Got non-200 response from smoke test.");
process.exit(1);
}
Loading
Loading