Skip to content

Align ToolCallAccuracy output schema to reason and harden skipped/properties handling#46355

Merged
m7md7sien merged 1 commit intomohessie/unify_output/tool_call_accuracyfrom
copilot/update-reasoning-to-reason
Apr 16, 2026
Merged

Align ToolCallAccuracy output schema to reason and harden skipped/properties handling#46355
m7md7sien merged 1 commit intomohessie/unify_output/tool_call_accuracyfrom
copilot/update-reasoning-to-reason

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 16, 2026

This update aligns the Tool Call Accuracy evaluator with the PR-expected response contract by renaming reasoning to reason across evaluator output, prompt schema, and scoped tests. It also corrects not-applicable payload shape (properties: null) and prevents runtime issues when LLM returns properties: null.

  • Contract updates in evaluator output

    • _return_not_applicable_result now emits:
      • tool_call_accuracy_reason (was tool_call_accuracy_reasoning)
      • tool_call_accuracy_properties: None (was {})
    • ToolCallAccuracyEvaluator now reads llm_output["reason"] and emits ..._reason.
  • Null-safe properties merge

    • Updated properties extraction to handle nullable payloads safely:
    llm_properties = llm_output.get("properties", {}) or {}
  • Prompt/schema alignment

    • tool_call_accuracy.prompty now uses reason (not reasoning) in skipped example and output-key instructions.
    • Output key order was updated so reason appears before score, matching requested schema ordering.
  • Scoped test updates for this PR surface

    • Updated relevant Tool Call Accuracy unit tests from reasoning/_reasoning to reason/_reason.
    • Kept changes limited to tests touched by this PR scope.

…ed properties handling

Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/89b3b528-f2ac-4284-88fb-c484d4c0cce1

Co-authored-by: m7md7sien <16615690+m7md7sien@users.noreply.github.com>
@m7md7sien m7md7sien marked this pull request as ready for review April 16, 2026 20:10
@m7md7sien m7md7sien requested a review from a team as a code owner April 16, 2026 20:10
@m7md7sien m7md7sien merged commit aa848fe into mohessie/unify_output/tool_call_accuracy Apr 16, 2026
1 check passed
@m7md7sien m7md7sien deleted the copilot/update-reasoning-to-reason branch April 16, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants