-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Problem
When running evals with the copilot-cli provider, file references in test inputs (e.g., type: file) are rejected by copilot's permission system. The copilot agent attempts to read the file but the tool call is denied, resulting in an empty answer and all assertions failing.
Repro
targets.yaml:
- name: copilot
provider: copilot
judge_target: azure-base
timeout_seconds: 180
log_format: jsoneval.yaml:
$schema: 'agentv-eval-v2'
description: Example eval
tests:
- id: example-test
criteria: |
Agent should respond based on the provided prompt file.
input:
- role: user
content:
- type: text
value: "Do something based on the prompt"
- type: file
value: /path/to/prompt-file.mdRun:
agentv eval eval.yaml --target copilotResult: Score 0.000 — copilot reads the file reference but the tool call is rejected:
{"event":"tool_call_update","data":{"rawOutput":{"message":"The user rejected this tool call.","code":"rejected"},"status":"failed"}}Workaround
Adding args: ["--yolo"] to the target config fixes it:
- name: copilot
provider: copilot
judge_target: azure-base
timeout_seconds: 180
log_format: json
args: ["--yolo"]Expected behavior
The copilot-cli provider should automatically pass --yolo (or at minimum --allow-all-paths) when launching copilot for evals, since eval workspaces are temporary and file access restrictions are counterproductive in this context.
Alternatively, agentv could use --add-dir to allowlist the eval workspace and repo root directories.