[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-16

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-16

Summary

Analysis Period: Last 30 days (2026-07-30 to 2026-08-16)
Total PRs: 1000 | Merged: 767 (76.7%) | Closed: 230 (23.0%) | Open: 3 (0.3%)

Note: The dataset is capped at 1000 PRs, so it represents the most recent ~1000 copilot/* PRs within the window, not necessarily every PR created.

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Feature Addition	300	241	80.3%
Other	22	18	81.8%
Bug Fix	469	356	75.9%
Documentation	61	45	73.8%
Testing	116	85	73.3%
Refactoring	32	22	68.8%

Categorized by keyword matching against PR title + body (bug/fix/error terms, add/implement/create terms, refactor/optimize terms, docs terms, test/coverage terms).

Prompt Analysis

✅ Successful Prompt Patterns

Merged PRs:

• Average prompt length: 151 words — noticeably shorter than closed PRs
• Frequent terms: workflow, run, actions, pkg, agent, test, engine, added — suggests concrete, implementation-focused language tied to specific files/subsystems
• Prompts tend to reference specific packages/files (pkg, engine) and describe a concrete before/after change

Example successful prompts:

1. PR Fix Linter Miner sub-agent model configuration to avoid Copilot model_not_supported failures #53166: "Linter Miner was failing in scheduled runs when delegating to linter-writer because the configured sub-agent model res..." → Merged
2. PR Update Serena MCP to the project-maintained image #53163: "The shared Serena MCP configuration was still pinned to the legacy GitHub-hosted image and binary entrypoint. That runti..." → Merged
3. PR Migrate 30 more agentic workflows to the gh-aw-detection feature #53162: "Continues the rollout of the external threat-detection engine by enabling features: gh-aw-detection: true on 30 additi..." → Merged

❌ Unsuccessful Prompt Patterns

Closed PRs:

• Average prompt length: 229 words — about 50% longer than merged PRs
• Frequent terms: cve, redacted, deb/debian, securitytracker, gpl, ubuntu — a large share of closed PRs are automated dependency/security-advisory PRs (e.g., CVE/package-tracker bumps) rather than task-driven Copilot prompts, which naturally have lower merge rates due to bulk/duplicate nature
• Some closed prompts open with exploratory, open-ended language ("Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a pla...") rather than a concrete task spec

Example unsuccessful prompts:

1. PR [WIP] Create /llms.txt for the documentation site #53158: "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a pla..." → Closed
2. PR Add formal test suite for AWF config drift escalation contract (P17-P19) #53148: "The formal model for specs/awf-config-sources-spec.md covered predicates P1-P16 but left Section 7.4.1's SLA escalatio..." → Closed
3. PR Reject digest-qualified container pin references #53140: "The CGO unit-test job failed because repository configuration accepted digest-qualified container pin keys and image val..." → Closed

Key Insights

• Conciseness correlates with success: Merged PR prompts average 151 words vs. 229 for closed PRs — roughly 35% shorter. Long, exploratory prompts appear to struggle more, possibly reflecting scope creep or ambiguous asks.
• Feature/Other prompts outperform Refactoring: Feature Addition (80.3%) and generic/Other (81.8%) categories have the highest merge rates, while Refactoring trails at 68.8% — refactors may be harder to validate/review or more prone to unintended side effects.
• Security/CVE-tracker PRs skew the closed-PR word list: A significant portion of closed PRs are automated CVE/dependency-tracking PRs (cve, redacted, debian, securitytracker, gpl terms dominate). These are likely closed by design/policy rather than reflecting poor Copilot prompt quality — worth excluding from future prompt-quality analysis for a cleaner signal.
• Open-ended "I will figure it out as I go" prompts underperform: Prompts lacking an upfront concrete plan or scope (e.g., "I will get started and keep this description up to date") appear among closed examples, suggesting clearer initial scoping helps.

Recommendations

1. DO: Keep prompts concise and specific — reference concrete files, packages, or subsystems (e.g., pkg/, specific config keys) rather than broad goals.
2. DO: Frame feature/bug-fix prompts around a clear before/after description of behavior, similar to the top merged examples.
3. AVOID: Very long, exploratory prompts (>200 words) without a concrete acceptance criterion — these correlate with lower merge rates.
4. AVOID / SEPARATE TRACKING: Automated CVE/dependency-tracker PRs should likely be excluded from prompt-quality analysis, since their close rate reflects triage policy, not prompt quality.

Historical Trends

Date	PRs	Merge Rate	Top Category
2026-08-16 (today)	1000	76.7%	Bug Fix (largest volume)
2026-07-07	1000	80.0%	Bug Fix
2026-07-06	1000	79.3%	Bug Fix
2026-07-05	1000	80.5%	Bug Fix
2026-07-04	1000	79.8%	Bug Fix
2026-07-03	1000	80.1%	Bug Fix
2026-07-01	—	81.2%	—

Trend: Merge rate has declined from roughly 79–81% in early July to 76.7% today, a drop of ~3-4 points. Average prompt length for merged PRs (151 words) is shorter than the ~189 words recorded on 2026-07-07, while closed-PR prompt length remains similarly high (~229 vs ~194 words), reinforcing the pattern that concise, specific prompts continue to correlate with merges even as overall volume and mix shift over time.

---

Generated by Copilot PR Prompt Analysis (Run: 31971353003)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 26.3 AIC · ⌖ 4.23 AIC · ⊞ 10.9K · ◷

• expires on Aug 17, 2026, 12:48 PM UTC-08:00