Skip to content

chore: make the tool-selection eval tolerant of Gemini's search-first pick - #95

Merged
WilliamBergamin merged 4 commits into
mainfrom
fix-broken-evals
Jul 16, 2026
Merged

chore: make the tool-selection eval tolerant of Gemini's search-first pick#95
WilliamBergamin merged 4 commits into
mainfrom
fix-broken-evals

Conversation

@WilliamBergamin

@WilliamBergamin WilliamBergamin commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request stops the nightly tool-selection eval from failing when the Gemini judge picks a search tool as its first step on an ambiguous prompt.

The evals are there to make sure skills are not selected over MCP tools, as long as these eval select an MCP tool then they achieve their goal

Preview

N/A — test-only change.

Testing

  • Run make test-eval and confirm tests/eval/test_tool_selection.py::TestToolSelection::test_tool_selection[ambiguous-reply-in-thread] passes.

Notes

  • Test-only change, so no changeset is included.

Requirements

@changeset-bot

changeset-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: fa3601a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@WilliamBergamin WilliamBergamin self-assigned this Jul 16, 2026
@WilliamBergamin
WilliamBergamin requested a review from vegeris July 16, 2026 15:58
@WilliamBergamin WilliamBergamin added the test Improve of update the tests of this project label Jul 16, 2026
@WilliamBergamin
WilliamBergamin marked this pull request as ready for review July 16, 2026 15:59
@WilliamBergamin
WilliamBergamin requested a review from a team as a code owner July 16, 2026 15:59

@vegeris vegeris left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me!

@WilliamBergamin
WilliamBergamin merged commit fa7996c into main Jul 16, 2026
7 checks passed
@WilliamBergamin
WilliamBergamin deleted the fix-broken-evals branch July 16, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Improve of update the tests of this project

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants