[aw-failures] Fix PR Sous Chef: Copilot CLI emits wrong safe-output field names (pr_number vs pull_request_number, pr/pr_number vs item_number [Content truncated due to length]

[github-actions]

Problem

PR Sous Chef (Copilot CLI engine) has a 37.5% failure rate over the last 6 hours (3 failures / 8 runs). In two of those three failures, the agent generated safe-output items with field names that don't match the MCP tool schema, so the safe-outputs validator rejected them and the workflow run was marked failure.

Affected Workflow & Runs

• Workflow: PR Sous Chef
• Failing runs in the 6h window ending 2026-05-14 01:25 UTC:
  • §25832686137 — update_pull_request field-name mismatch
  • §25831659910 — add_comment field-name mismatch (2 messages)
  • §25824436959 — not a field-name issue; correct fields but transient GitHub API error on PR update (out of scope for this sub-issue)

Probable Root Cause

The Copilot CLI is hallucinating REST-API-style field names instead of using the MCP tool schema's actual field names:

MCP tool	Expected field	Generated field
update_pull_request	pull_request_number (number)	pr_number (number)
add_comment (PR target)	item_number or issue_number (number)	pr_number (string) or pr (string)

Validator error messages from the workflow logs:

##[error]✗ Message 1 (update_pull_request) failed: Target is "*" but no pull_request_number specified in update_pull_request item
##[error]✗ Message 1 (add_comment) failed: Target is "*" but no item_number/issue_number specified in add_comment item
##[error]✗ Message 2 (add_comment) failed: Target is "*" but no item_number/issue_number specified in add_comment item

Sample agent_output.json content from the failing runs:

{"body":"Updating branch to resolve mergeability.","pr_number":32012,"update_branch":true,"type":"update_pull_request"}
{"body":"Quick nudge: `@copilot` please summarize...","pr_number":"32015","type":"add_comment"}
{"body":"pr-sous-chef: quick nudge ...","pr":"32012","type":"add_comment"}

Proposed Remediation

Pick one (or combine):

1. Strengthen the PR Sous Chef system prompt — add an explicit "Field names" section enumerating the exact field names for update_pull_request and add_comment, with copy-paste-ready JSON examples. The model appears to be falling back to REST API naming conventions.
2. Add field-name aliasing in the safe-outputs validator (defensive, repo-wide benefit) — accept pr_number, pr, pull_request_id, pullRequestNumber, etc. as synonyms for pull_request_number / item_number when the tool target is unambiguously a PR. This protects all Copilot-engine workflows from the same class of bug.
3. Tighten MCP tool schema descriptions — explicitly call out "Use pull_request_number, NOT pr_number" in the update_pull_request and add_comment tool descriptions surfaced to the model.

Recommendation: do (1) immediately (low-risk, fast) and consider (2) as a defense-in-depth follow-up.

Success Criteria

• PR Sous Chef failure rate drops below 5% over a rolling 24h window.
• No safe-output validator errors of the form no pull_request_number specified or no item_number/issue_number specified for PR Sous Chef over 7 consecutive days.

Verification

After the fix lands:

1. Re-run PR Sous Chef twice on its normal cron cadence.
2. Confirm no Failed: N lines appear in the safe_outputs job logs.
3. Confirm agent_output.json items consistently use the correct field names.

Parent: #32039
Related to #32039

Generated by [aw] Failure Investigator (6h) · ● 17.6M · ◷

• expires on May 21, 2026, 1:35 AM UTC

[pelikhan]

@copilot the tools.json schema should support aliases for parameter names that are commonly confused by agents/models, like pr_number or pr in this case.