[evaluation] Enable groundedness for Bing and OpenAPI tool calls - #48304
Closed
mmkawale wants to merge 1 commit into
Closed
[evaluation] Enable groundedness for Bing and OpenAPI tool calls#48304mmkawale wants to merge 1 commit into
mmkawale wants to merge 1 commit into
Conversation
Align GroundednessConversationValidator with assets policy: allow bing_grounding, bing_custom_search, and openapi_call in addition to already-enabled azure_ai_search/azure_fabric/sharepoint_grounding; retain guardrails for browser_automation/code_interpreter_call/computer_call/web_search. Update groundedness validator comments and validator unit tests to reflect the narrowed unsupported list. Validation: black on touched files; PYTHONPATH=sdk/evaluation/azure-ai-evaluation pytest tests/unittests/test_common_validators.py -k 'UnsupportedToolsListGroundednessValidator or UnsupportedToolsListConversationValidator or ConversationValidatorRejectsStillUnsupportedTools' (24 passed).
github-actions
Bot
added
the
Evaluation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Enables additional groundedness tool-call validation while retaining guardrails for unsupported tools.
Changes:
- Narrows Groundedness’s unsupported-tool list.
- Updates validator tests and evaluator comments.
- Applies Black formatting to touched code.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
_conversation_validator.py |
Revises Groundedness tool restrictions. |
_groundedness.py |
Updates evaluator comments and formatting. |
test_common_validators.py |
Updates supported-tool validation tests. |
Comment on lines
+531
to
+534
| UNSUPPORTED_TOOLS: List[str] = [ | ||
| tool_name | ||
| for tool_name in ConversationValidator.UNSUPPORTED_TOOLS | ||
| if tool_name not in {"bing_custom_search", "bing_grounding", "openapi_call"} |
Comment on lines
+123
to
+125
| # keeps stricter guardrails than the shared base for the remaining | ||
| # unsupported tool families, while allowing grounding/search/openapi | ||
| # tool calls supported by this evaluator. |
Comment on lines
+761
to
+764
| @pytest.mark.parametrize( | ||
| "tool_name", NEWLY_ENABLED_TOOLS + GROUNDEDNESS_NEWLY_ENABLED_TOOLS | ||
| ) | ||
| def test_groundedness_validator_accepts_newly_enabled_tools(self, tool_name): |
Contributor
Author
|
Superseded by #48305, which contains the same groundedness behavior updates with significantly reduced whitespace/formatting churn for easier review. |
Contributor
Author
|
Closing in favor of #48305 (clean replacement with minimal diff noise). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bing_groundingbing_custom_searchopenapi_callazure_ai_searchazure_fabricsharepoint_groundingbrowser_automationcode_interpreter_callcomputer_callweb_searchChanges
GroundednessConversationValidator.UNSUPPORTED_TOOLSto align with current groundedness policytest_common_validators.pyto validate the new list and behaviorValidation
python -m blackon touched filesPYTHONPATH=sdk/evaluation/azure-ai-evaluation python -m pytest sdk/evaluation/azure-ai-evaluation/tests/unittests/test_common_validators.py -k "UnsupportedToolsListGroundednessValidator or UnsupportedToolsListConversationValidator or ConversationValidatorRejectsStillUnsupportedTools"24 passed