Add tool call assertions to Agent Evals

[samedson]

Description

• Adds a new method expectToolCalls that takes a list of tool names and asserts they were called since the last turn. Borrows this implementation from the Gemini CLI's test rig
• Also unrelatedly, organizes the output folder by when the test was started, for better organization

Scenarios Tested

You can assert that the tool was called with arguments and what its success was:

  it("backend app", async function (this: Mocha.Context) {
    const run: AgentTestRunner = await startAgentTest(this);

    await run.type("/firebase:init");
    await run.expectText("Backend Services");
    await run.expectText("AI Logic");

    await run.type(
      "Build a single page backend app with html and pure javascript. It should say Hello World, but let you login and edit the hello world text for your user",
    );

    await run.type("Yes that looks good. Use Firebase Project gcli-ext-sam-01");
    await run.expectToolCalls([
      "firebase_update_environment",
      {
        name: "firebase_read_resources",
        argumentContains: "firebase://guides/init/backend",
        successIs: true,
      },
    ]);
  });

Asserted that this test fails, because firebase_update_environment was called in the first turn, but not the second:

  it.only("testing testing sam", async function (this: Mocha.Context) {
    const run: AgentTestRunner = await startAgentTest(this);

    await run.type("Call firebase_update_environment with gcli-ext-sam-01");
    await run.expectToolCalls([
      "firebase_update_environment",
    ]);

    await run.type(
      "Call firebase_list_apps I wanna see them",
    );
    // This will fail because only firebase_list_apps was called this turn
    await run.expectToolCalls([
      "firebase_update_environment",
    ]);
  });

[gemini-code-assist]

Summary of Changes

Hello @samedson, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the agent evaluation framework by providing a robust mechanism to assert tool calls made by the agent during a test run. This allows for more precise and reliable testing of agent behavior. Additionally, it refines the test output structure for improved clarity and maintainability.

Highlights

• New Tool Call Assertion: Introduced a new expectToolCalls method in the AgentTestRunner interface and its implementation, allowing tests to assert that specific tools were called by the agent since the last turn.
• Output Folder Organization: Improved the organization of test output folders by grouping them by the date the test run was started, making it easier to manage and review test results.
• Enhanced Error Reporting: Added a throwFailure utility function that logs error messages in a distinct color before throwing an error, providing better visibility for test failures in the console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable expectToolCalls assertion for agent evaluations and improves test output organization by using timestamped directories. The implementation is solid, but I've identified a couple of high-severity issues that should be addressed before merging: a potential runtime error from an unsafe type assertion and the use of it.only in a test file, which would prevent the full test suite from running. I've also included a suggestion to improve the efficiency of the new assertion logic.