Python: [Bug]: FoundryEvals MissingRequiredDataMapping for tool_definitions on task_adherence and other agent evaluators

[gowda64]

Description

When using evaluate_agent() with default evaluators on an agent that has tools, the Foundry evals service returns a 400 BadRequestError with MissingRequiredDataMapping.

What happened?
Calling evaluate_agent(agent=agent, queries=[...], evaluators=evals) where evals = FoundryEvals(client=chat_client) (default evaluators) fails when the agent has tools.

What did you expect to happen?
The evaluation should complete successfully. The tool_definitions data mapping should be included for all evaluators that support it, not just the ones in _TOOL_EVALUATORS.

Steps to reproduce:

1. Create an agent with tools
2. Create FoundryEvals(client=chat_client) with no explicit evaluators (uses defaults: relevance, coherence, task_adherence + auto-added tool_call_accuracy)
3. Call evaluate_agent(agent=agent, queries=[...], evaluators=evals)

Root cause: In agent_framework_foundry/_foundry_evals.py, _TOOL_EVALUATORS only contains 5 explicit tool evaluators but the service requires the tool_definitions data mapping for all agent evaluators that support it (task_adherence, task_completion, intent_resolution, task_navigation_efficiency) when the data contains that field.

Suggested fix: Add the missing agent evaluators to _TOOL_EVALUATORS:

_TOOL_EVALUATORS: set[str] = {
    "builtin.intent_resolution",
    "builtin.task_adherence",
    "builtin.task_completion",
    "builtin.task_navigation_efficiency",
    "builtin.tool_call_accuracy",
    "builtin.tool_selection",
    "builtin.tool_input_accuracy",
    "builtin.tool_output_utilization",
    "builtin.tool_call_success",
}

Code Sample

from agent_framework import Agent, evaluate_agent
from agent_framework.foundry import FoundryChatClient, FoundryEvals
from azure.identity import AzureCliCredential

chat_client = FoundryChatClient(
    project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    model="gpt-4o",
    credential=AzureCliCredential(),
)

def get_weather(location: str) -> str:
    return "62F, cloudy"

agent = Agent(client=chat_client, tools=[get_weather], name="test")
evals = FoundryEvals(client=chat_client)  # default evaluators

# Fails with MissingRequiredDataMapping
results = await evaluate_agent(agent=agent, queries=["What is the weather?"], evaluators=evals)

Error Messages / Stack Traces

openai.BadRequestError: Error code: 400 - {'error': {'code': 'UserError', 'message': 'Evaluation failed validation: {
  Code: MissingRequiredDataMapping
  Target: group.testing_criteria[2].data_mapping
  Message: Inline data contains field 'tool_definitions' which is supported by evaluator 'builtin.task_adherence' but is not included in the data_mapping. Add '"tool_definitions": "{{item.tool_definitions}}"' to the data_mapping to pass this field to the evaluator.
}'}}

Package Versions

agent-framework: 1.4.0, agent-framework-foundry: 1.4.0

Python Version

Python 3.12.10

Additional Context

The sample python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py (Pattern 2a) triggers this bug. Azure AI Foundry project region: eastus.