[copilot-opt] opt: scope-drift in closed Copilot PRs signals missing acceptance criteria in workflow prompts

[github-actions]

Problem

Multiple closed-without-merge Copilot PRs show scope drift: the implementation delivered does not match the original task description. Evidence includes PRs where the branch name references one change but the PR title and body describe a different change (e.g., branch copilot/investigate-safe-output-issue-again delivers a chaos-fuzzer fix; branch copilot/fix-otel-export-error delivers an OTLP shared-import alignment). This indicates the agent reinterprets the issue mid-session and pivots to a related but distinct change, causing maintainers to close the PR as a scope mismatch.

Evidence

• Analysis window: 2026-05-01 to 2026-05-18
• Sessions analyzed: 185 closed-unmerged PRs across 1,000 total Copilot PRs
• Key metrics and examples:
  • PR #32769 branch name: investigate-safe-output-issue-again; title: fix chaos-fuzzer branch-exists push failure (scope mismatch)
  • PR #32791 branch name: fix-otel-export-error; title: Align OTLP shared import with header-based secret contract (scope mismatch)
  • PR #32582 labeled spam; branch: update-claude-code-and-mcp-gateway — agent regenerated all compiled artifacts unnecessarily, triggering a spam close
  • 8 retry-suffixed branches found among closed PRs (-again, -yet-again, -one-more-time, -retry) — each suffix indicates the agent knew a prior attempt failed but lacked explicit acceptance criteria to course-correct

Proposed Change

1. Add a structured Acceptance Criteria section to each agentic workflow prompt that lists exactly what files must change, what tests must pass, and what reviewer questions must be answered.
2. Add a prompt instruction: Before opening a PR, confirm the branch name, the changed files, and the issue title all describe the same scope. If they diverge, stop and call report_incomplete.
3. For version-bump workflows specifically, add an explicit constraint: only update the target version constant and recompile — do not regenerate unrelated artifacts.

Expected Impact

• Reduce scope-mismatch closes (estimated 15-20% of the 185 closed-unmerged PRs based on title/branch mismatches observed)
• Reduce spam-labeled closes by constraining artifact regeneration scope
• Reduce retry suffixes by giving agents a clear self-check before pushing

Notes

• Distinct root cause category: prompt design — missing acceptance criteria / scope guard instructions
• Data quality: session event logs (events.jsonl) were not available for prior sessions; scope analysis is based on PR title vs. branch name divergence observed manually across 50 sampled closed PRs

Generated by ⚡ Copilot Opt · ● 16.7M · ◷