Feat/OpenAI agent builtin tools 6657 #6671
Conversation
…on guide- Fix version format from 0.4.0-dev-1 to 0.4.0-dev.1 for all packages- Remove reference to non-existent Microsoft.AutoGen.Extensions package- Add correct extension packages: Aspire, MEAI, and SemanticKernel- Fix typo: RuntimeGatewway -> RuntimeGateway- Improve documentation structure with clear section headersFixes microsoft#6244
Fix issue microsoft#6277 where TextMessage was used but not imported in three code cells of the custom agents documentation, causing NameError when users run the examples. Changes: - Add TextMessage to imports in ArithmeticAgent section - Add TextMessage to imports in GeminiAssistantAgent section - Add TextMessage to imports in Declarative GeminiAssistantAgent section The CountDownAgent section already had the correct import. Fixes microsoft#6277
- Support file_search, code_interpreter, web_search_preview, computer_use_preview, image_generation, and mcp tools - Add comprehensive tests for all tool types including mixed usage scenarios - Include working examples demonstrating tool usage - Update documentation with version notes and usage examples - Maintain backward compatibility with existing function tools Fixes microsoft#6657
|
@withsmilo Would you mid providing a first review here? |
|
Sure. Let me review today night. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #6671 +/- ##
==========================================
+ Coverage 79.67% 79.69% +0.01%
==========================================
Files 231 231
Lines 17172 17188 +16
==========================================
+ Hits 13682 13698 +16
Misses 3490 3490
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
docs/dotnet/core/installation.md
Outdated
| dotnet add package Microsoft.AutoGen.AgentChat --version 0.4.0-dev-1 | ||
| dotnet add package Microsoft.AutoGen.Agents --version 0.4.0-dev-1 | ||
| dotnet add package Microsoft.AutoGen.Extensions --version 0.4.0-dev-1 | ||
| dotnet add package Microsoft.AutoGen.AgentChat --version 0.4.0-dev.1 |
There was a problem hiding this comment.
@tejas-dharani ,
Changes should be scoped to the PR title/focus
This file is unrelated to the OpenAI agent
There was a problem hiding this comment.
Hi @victordibia,
You're absolutely right - my apologies for the confusion.
My mistake was that after forking the repo, I raised my first PR (the .NET correction) directly from the main branch instead of creating a separate branch. After that, I created separate branches for all other issues, but now this .NET change is affecting every subsequent PR since they're all based off my modified main branch.
I understand this file is unrelated to the OpenAI agent work. I'll close the .NET PR to keep it separate from the OpenAI agent changes.
Thanks for catching this - lesson learned about proper Git workflow!
| @@ -0,0 +1,231 @@ | |||
| """ | |||
There was a problem hiding this comment.
Ideally, this file should be removed.
Please follow convention - examples should be in the docstrings
| Iterable[ | ||
| Union[ | ||
| Literal[ | ||
| "file_search", |
There was a problem hiding this comment.
Hi @tejas-dharani .
According to the official doc, the Responses API supports 7 built-in tools as well as function calls. However, I found your PR omitted one built-in tool (local_shell) and lots of "required" parameters to execute built-in tools. Could you please check the official doc in detail once more?
-
file_search
- vector_store_ids (required)
- filters
- max_num_results
- ranking_options
-
web_search_preview
- search_context_size
- user_location
-
computer_use_preview
- display_height (required)
- display_width (required)
- environment (required)
-
mcp
- server_label (required)
- server_url (required)
- allowed_tools
- headers
- require_approval
-
code_interpreter
- container (required)
-
image_generation
- background
- input_image_mask
- ....
-
local_shell
There was a problem hiding this comment.
Hi @withsmilo,
Thank you for the detailed feedback! I missed required parameters for each tool type.
I'll address:
Implement all required parameters (vector_store_ids, display dimensions, container, server configs, etc.)
Add proper parameter validation and type hints
Fix the failing mypy/pyright checks
Will research the official docs more thoroughly for the complete specifications.
Thanks for the thorough review!
There was a problem hiding this comment.
@withsmilo
I don't find the local_shell , are you refering to function calling ? Function calling is already there.
https://platform.openai.com/docs/guides/tools?api-mode=responses
There was a problem hiding this comment.
Hi @tejas-dharani , I compared the available tools present in the openai docs with the ones added in the PR. What I think @withsmilo is mentioning is that the local shell tool has not been implemented.
There was a problem hiding this comment.
I see now, local shell tool is missing in Available tools openai docs.
Also Local shell is available through the Responses API for use with codex-mini-latest. It is not available on other models.
…feat/openai-agent-builtin-tools-6657
….com/tejas-dharani/autogen into feat/openai-agent-builtin-tools-6657
…stallation guide- Fix version format from 0.4.0-dev-1 to 0.4.0-dev.1 for all packages- Remove reference to non-existent Microsoft.AutoGen.Extensions package- Add correct extension packages: Aspire, MEAI, and SemanticKernel- Fix typo: RuntimeGatewway -> RuntimeGateway- Improve documentation structure with clear section headersFixes microsoft#6244" This reverts commit 6d9fb2e.
…tation" This reverts commit d07898b.
….com/tejas-dharani/autogen into feat/openai-agent-builtin-tools-6657
Why are these changes needed?
This PR adds comprehensive built-in tool support to OpenAIAgent, enabling support for file_search, code_interpreter, web_search_preview, computer_use_preview, image_generation, and mcp tools. The changes include comprehensive testing for all tool types including mixed usage scenarios, working examples demonstrating tool usage, updated documentation with version notes and usage examples, while maintaining backward compatibility with existing function tools.
Related issue number
Fixes #6657
Checks