Improve tool selection with additional tool descriptions#1718
Improve tool selection with additional tool descriptions#1718ravikiranvm merged 3 commits intomainfrom
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
|
There was a problem hiding this comment.
Pull request overview
This PR improves LLM tool selection by adding supplementary descriptions for external tools. The changes help the AI distinguish between tools with overlapping functionality, specifically preventing the misuse of the 'session-sql' tool (designed for AWS billing queries) when users ask about OpenOps tables.
Key Changes:
- Added a system for providing additional tool usage notes to guide LLM tool selection
- Integrated these notes into both the query router and main chat prompts
- Created initial tool descriptions to clarify 'session-sql' tool scope limitations
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
external-tool-descriptions-data.ts |
Defines tool-specific usage notes and metadata structure |
external-tool-descriptions.ts |
Implements helper function to extract relevant tool descriptions |
llm-query-router.ts |
Integrates additional tool notes into the system prompt for tool routing |
prompts.service.ts |
Adds additional tool descriptions to the main chat system prompt |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Greptile OverviewGreptile SummaryThis PR adds a mechanism to inject additional tool usage notes into system prompts for both the LLM query router and MCP chat flows. This addresses tool selection confusion by providing explicit guidance about when specific tools should or should not be used.
The implementation follows existing codebase patterns and cleanly separates data from logic. Confidence Score: 5/5
Important Files ChangedFile Analysis
Sequence DiagramsequenceDiagram
participant User
participant QueryRouter as llm-query-router
participant PromptsService as prompts.service
participant ToolDescriptions as external-tool-descriptions
participant ToolData as external-tool-descriptions-data
Note over User,ToolData: Tool Selection Flow
User->>QueryRouter: Send query with tools
QueryRouter->>ToolDescriptions: getAdditionalToolDescriptions(toolNames[])
ToolDescriptions->>ToolData: lookup MCP_TOOL_ADDITIONAL_DESCRIPTIONS
ToolData-->>ToolDescriptions: matching descriptions
ToolDescriptions-->>QueryRouter: string[] of notes
QueryRouter->>QueryRouter: append to system prompt
Note over User,ToolData: MCP Chat Flow
User->>PromptsService: getMcpSystemPrompt(selectedTools)
PromptsService->>ToolDescriptions: getAdditionalToolDescriptions(ToolSet)
ToolDescriptions->>ToolData: lookup MCP_TOOL_ADDITIONAL_DESCRIPTIONS
ToolData-->>ToolDescriptions: matching descriptions
ToolDescriptions-->>PromptsService: string[] of notes
PromptsService->>PromptsService: buildAdditionalToolNotes
PromptsService-->>User: complete system prompt
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -0,0 +1,15 @@ | |||
| export type ToolDescription = { | |||
| note: string; | |||
| mcpServer?: string; | |||
There was a problem hiding this comment.
style: The mcpServer field is defined but not used anywhere in the codebase. Consider whether this is needed for future functionality or should be removed to keep the interface minimal.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/server/api/src/app/ai/mcp/external-tool-descriptions-data.ts
Line: 3:3
Comment:
**style:** The `mcpServer` field is defined but not used anywhere in the codebase. Consider whether this is needed for future functionality or should be removed to keep the interface minimal.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Added for future functionality in case if we would need it
| const toolsMessage = toolList | ||
| .map((t) => `- ${t.name}: ${t.description}`) | ||
| .join('\n'); | ||
| const additionalToolNotes = getAdditionalToolDescriptions( |
There was a problem hiding this comment.
I'm not quite sure about this. Since we are sending all tools to LLM for tool selection, this would also send all additional tool descriptions along with the tool selection system prompt.
Fixes OPS-3138.