[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-20

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-20

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 770 (77.0%) | Closed: 213 (21.3%) | Open: 17 (1.7%)

Note: 1000 is the fetch cap — the actual PR volume for this window may be higher, so percentages are computed over the sampled set.

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Refactoring	16	14	87.5%
Bug Fix	233	162	69.5%
Testing	71	54	76.1%
Feature Addition	83	61	73.5%
Documentation	48	35	72.9%
Other/Uncategorized	549	444	80.9%

Categorization is keyword-based on the first ~300 characters of each PR body; ~55% of PRs didn't match a specific bucket ("Other"), which skews the overall rate upward since many "Other" prompts are actually well-scoped agentic-workflow tasks (e.g., spec reviews, config changes) rather than vague requests.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: ~188 words (shorter than closed)
• Most common keywords (excluding boilerplate like "github", "com", "https"): workflow, run, actions, copilot, agent, gpt
• Refactoring prompts had the highest success rate (87.5%), followed by "Other" (80.9%) and Testing (76.1%)

Example successful prompts:

1. PR Skip GHES auto-detection during CI initialization #54112: "GHES detection during init can be misled by gh-proxy environment settings and must not run in CI. The repository sho..." → Merged
2. PR Enable external threat detection by default #54111: "gh-aw-detection should default to the external threat-detection implementation while retaining a feature flag for lega..." → Merged
3. PR [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #54109: "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a pla..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: ~227 words (~21% longer than merged prompts)
• Keyword issue and cve appear disproportionately more often relative to merged prompts, suggesting issue-triage/security-flag-driven PRs close more often (possibly false positives or superseded by other fixes)
• Bug Fix had the lowest success rate of the specific categories (69.5%), despite being the most common category (233 PRs)

Example unsuccessful prompts:

1. PR Add Claude engine examples to reference documentation #54099: "Engine-agnostic reference pages presented Copilot-only examples, leaving Claude users without copyable workflows and exp..." → Closed
2. PR Sync spec/compliance docs: CR-06a fixture row, CTR ID rationale, forecast fixture check, cron edge case #54044: "Daily SPDD spec review of awf-config-sources-*, compiler-threat-detection-*, and forecast-compliance-fixtures flag..." → Closed
3. PR [WIP] Fix safe-outputs.runs-on type mismatch #53989: "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a pla..." → Closed

Key Insights

• Length matters: Closed PR prompts average ~227 words vs. ~188 for merged — the longest 20% of prompts correlate with lower merge rates, suggesting overly broad or multi-part asks are harder for the agent to land cleanly. Favor a single, well-scoped change per prompt.
• Bug fixes underperform relative to volume: Bug Fix is the largest category (233 PRs) but has the lowest success rate (69.5%) among named categories. Prompts mentioning issue/cve keywords skew toward closures — likely tied to automated security/triage prompts that get superseded, deduplicated, or rejected after review rather than genuinely broken code.
• Refactoring prompts convert best: 87.5% success rate, though on a small sample (16 PRs) — narrowly-scoped "improve X" tasks tend to be low-risk, easy-to-review changes that merge cleanly.
• Template/boilerplate openers ("Thanks for asking me to work on this...") appear in both merged and closed examples — the opening framing itself isn't predictive; what follows (specificity of the ask) likely determines outcome.

Recommendations

Based on today's analysis:

1. DO: Keep prompts focused on a single, concrete change (target ~150-200 words) rather than bundling multiple asks — shorter prompts trend toward higher merge rates.
2. DO: For refactoring/cleanup tasks, be explicit about the specific function/file/pattern to improve — this category has the best conversion rate.
3. AVOID: Framing bug-fix prompts purely around an issue/CVE reference without also specifying the exact reproduction steps or expected behavior — these tend to close more often, possibly due to being superseded by manual fixes or found to be non-issues.

Historical Trends

Date	PRs	Success Rate	Top Category
2026-08-20	1000	78.3%	bug_fix
2026-07-06	1000	80.4%	test
2026-07-05	1000	80.8%	bug_fix
2026-07-04	1000	81.1%	bug_fix
2026-07-03	1000	81.8%	bug_fix

Trend: Success rate has drifted down from ~81-82% in early July to 78.3% today (note: there's a data gap between 2026-07-06 and this run, so the trend covers a wider window than the usual daily cadence). Bug Fix remains the dominant top category across most snapshots, consistent with today's finding that it's the highest-volume but comparatively lower-converting category.

---

Generated by Copilot PR Prompt Analysis (Run: 32415947034)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 20.6 AIC · ⌖ 6.67 AIC · ⊞ 11.1K · ◷

• expires on Aug 21, 2026, 12:52 PM UTC-08:00