[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-08

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-08

Summary

Analysis period: last 30 days (2026-07-09 → 2026-08-08), 1000 PRs sampled from copilot/* branches (dataset cap; true volume may be higher).

Total: 1000 | Merged: 744 (74.4%) | Closed: 253 (25.3%) | Open: 3 (0.3%)

Prompt categories and success rates

Category	Total	Merged	Success Rate
Testing	45	36	80.0%
Other	98	79	80.6%
Documentation	30	23	76.7%
Feature Addition	250	187	74.8%
Refactoring	38	28	73.7%
Bug Fix	539	391	72.5%

Categorized by keyword matching on title+body; "Other" catches PRs without a clear keyword match.

Prompt analysis

Length: Merged prompts average 192 words; closed prompts average 244 words — a ~27% longer average for closed PRs, consistent with prior runs (189 vs 194 on 2026-07-07, a smaller gap that has widened).

Word frequency detail

Top words in merged prompt bodies: github, workflow, sous/chef (agent branding tokens), com, https, run, actions, runs, copilot, engine.

Top words in closed prompt bodies: github, cve, redacted, workflow, org, deb/debian, securitytracker, fix, gpl, high.

The closed-PR vocabulary is dominated by CVE/security-advisory scanning terms (cve, redacted, debian, gpl, securitytracker, high) — strongly suggesting a batch of automated dependency/security-scan PRs makes up a large share of closures, likely duplicates, false positives, or low-priority findings that maintainers reject rather than merge.

✅ Successful prompt patterns

• Concise, scoped tasks: merged prompts run ~50 words shorter on average than closed ones.
• Concrete change description with a diff/example: top merged examples cite specific files (specs/aw-harness.md), specific workflows, and include a ### Changes / example diff block.
• Testing and misc/maintenance prompts have the highest success rates (80%), likely because they are narrowly scoped and low-risk to review.

Example successful prompts:

1. PR Import shared reporting guidance into workflows missing report formatting rules #51431: "Daily style review flagged reporting workflows whose prompts lack markdown formatting guidance..." → Merged
2. PR remove aw-harness spec and all references #51428: "Deletes specs/aw-harness.md and removes every reference to it across the repo." → Merged
3. PR Allow Smoke Copilot dispatches to main #51416: "Smoke Copilot's safe_outputs job rejected its configured dispatch_workflow call..." → Merged

❌ Unsuccessful prompt patterns

• CVE/security-scan boilerplate: many closed PRs are auto-generated from dependency vulnerability scans (cve, securitytracker, debian, gpl boilerplate) — these read as templated, low-context reports rather than actionable, reviewed changes.
• Longer, more diffuse prompts: closed prompts are on average ~52 words longer, suggesting broader/less-focused scope or duplicated advisory text bloating the body.
• Open-ended "I will get started and keep this updated" prompts (PR [WIP] Add robots.txt to docs site allowing AI crawlers #51419 style) signal an in-progress/placeholder body rather than a completed, reviewable change — a red flag correlating with closure.

Example unsuccessful prompts:

1. PR Verify aw-compat audit findings (#51044) are already resolved — no code changes required #51424: "The daily cross-repo compatibility audit ([aw-compat] Daily AW compatibility audit: 20 repos, 2 unresolved compile failures, codemods validated in the wild #51044) flagged two actionable gaps..." → Closed
2. PR Soften Design Decision Gate invocation-cap failures #51422: "Design Decision Gate was capped at 20 LLM invocations, causing complex ADR reviews to fail..." → Closed
3. PR [WIP] Add robots.txt to docs site allowing AI crawlers #51419: "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date..." → Closed (placeholder/WIP body)

Key insights

• Pattern 1: Bug-fix prompts are the single largest category (539 of 1000) but have the lowest success rate (72.5%) — likely inflated by a wave of low-value automated CVE/security-scan PRs that get closed.
• Pattern 2: Prompt length inversely correlates with success — merged PRs are noticeably more concise (192 vs 244 words), reinforcing that focused, single-purpose prompts merge more often than sprawling ones.
• Pattern 3: Placeholder/"I will keep this updated" bodies are a leading indicator of closure — prompts should describe a completed change with concrete diffs/examples, not a future intent.

Recommendations

1. DO: Keep prompts concise (~150-200 words) and reference specific files/workflows/functions being changed.
2. DO: Include a concrete ### Changes section or diff snippet showing exactly what was modified, as seen in the highest-success examples.
3. AVOID: Submitting placeholder/in-progress prompt bodies ("I will get started...") — wait until the change is complete and describe the actual diff.
4. AVOID / TRIAGE: Auto-generated CVE/security-scan prompts appear to have a much lower merge rate; consider filtering or hand-triaging these before opening a PR, or add explicit remediation reasoning instead of raw advisory dumps.

Historical trends

Date	PRs	Merge Rate	Notes
2026-08-08 (today)	1000	74.4%	Bug Fix largest category, lowest rate (72.5%)
2026-07-07	1000	80.0%	Bug Fix 80.8% rate
2026-07-06	1000	80.4%	Top category: test
2026-07-05	1000	80.8%	—
2026-07-04	1000	79.8%	—
2026-07-03	1000	81.8%	—

Trend: Merge rate has declined ~6 points since early July (80.8% → 74.4%), coinciding with a larger share of Bug Fix category PRs and a widening gap between merged/closed prompt length. This is consistent with more automated/security-scan-sourced PRs entering the pipeline with lower merge rates.

---

Generated by Copilot PR Prompt Analysis (Run: 31277742167)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 21 AIC · ⌖ 4.81 AIC · ⊞ 10.8K · ◷

• expires on Aug 9, 2026, 12:52 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-09T20:52:22.060Z.

Closed by Workflow