[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-08

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-08

Summary

Analysis Period: Last 30 days (2026-08-17 to 2026-09-08)
Total PRs: 1000 | Merged: 808 (80.8%) | Closed: 188 (18.8%) | Open: 4 (0.4%)
Merge rate (completed PRs): 81.1%

Note: the dataset was capped at 1000 PRs (the fetch limit), so this window covers a very high-volume ~3-week slice rather than a strict 30-day sample.

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Bug Fix	783	654	125	84.0%
Feature Addition	177	123	54	69.5%
Other	25	18	7	72.0%
Documentation	6	4	2	66.7%
Testing	7	7	0	100%
Refactoring	2	2	0	100%

Categorization is keyword-based (title + body) and categories overlap in practice, so bug-fix-flavored language dominates volume in this repo's task mix.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 181.7 words (notably shorter than closed PRs)
• Code blocks present: 83.5% of merged PR bodies
• Bullet/structured lists: 97.0% of merged PR bodies
• "WIP"/"do not merge" markers: only 0.4%
• Top recurring terms: github, workflow, run, actions, copilot, agent, model, engine, generated

Example successful prompts:

1. PR Resolve remote workflow names for logs targets #59454: "Remote logs targets used the agentic workflow ID directly with gh run list, which fails when the GitHub Actions workflow has a different display name..." → Merged
2. PR Add oldest-run pruning mode to logs downloads #59453: "Adds an opt-in mode that removes older cached runs when --max-storage cannot be satisfied by pruning non-essential data..." → Merged
3. PR Relocate Claude debug log outside agent data directory #59450: "Claude's debug log was stored under /tmp/gh-aw/agent, which is reserved for user-generated data. This could mix engine..." → Merged

These share a consistent shape: problem statement first ("X was broken because Y"), followed by a short bulleted change list, referencing specific files/commands/flags.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 308.0 words — nearly 70% longer than merged prompts
• Code blocks present: only 65.4% of closed PR bodies (18pp lower than merged)
• "WIP"/"do not merge" markers: 1.1% (~3x more common than merged)
• Top recurring terms: operational, value, evidence, study, metric, null, graders, blocked — language suggestive of exploratory/analytical or evaluation-style tasks rather than concrete code changes

Example unsuccessful prompts:

1. PR Stream large Copilot prompts through stdin #59312: "Threat detection replaced prompts over 100KB with a compact file-reading instruction, causing immediate Copilot backend..." → Closed
2. PR Resolve extensionless logs workflow targets #59254: "gh aw logs could not resolve extensionless targets for raw GitHub Actions workflows. Extensionless targets now prefer..." → Closed
3. PR Enable strict mode for all workflow compilations #59246: "Enforce strict validation across all repository gh aw compile invocations. ## Changes - Enable repository-wide strict..." → Closed

Note: some closed examples above look structurally similar to successful ones (bullets, code refs) — length and scope crept were the bigger differentiators; several closed PRs' bodies referenced broad, multi-part validation/reporting efforts (operational, evidence, metric, graders) rather than a single, bounded fix.

Key Insights

• Conciseness correlates with merge success: merged prompts average 182 words vs. 308 for closed prompts — nearly double. Long prompts likely signal broader, harder-to-review, or under-specified scope.
• Code-block usage matters: 83.5% of merged prompts include a fenced code block (error message, snippet, or command) vs. 65.4% of closed ones — grounding the task in concrete code/output appears to improve outcomes.
• "WIP"/exploratory language is a red flag: prompts flagged with WIP or "do not merge" markers are ~3x more common among closed PRs (1.1% vs 0.4%), and closed-PR vocabulary skews toward evaluative/analytical terms (operational, evidence, metric, study) rather than direct fixes.
• Bounded categories perform best: Testing (100%) and Refactoring (100%, small n=2) had the highest success rates, while Feature Addition (69.5%) and Documentation (66.7%) lagged behind Bug Fix (84.0%) — likely because bug-fix prompts tend to describe a single, concrete, reproducible problem.

Recommendations

1. DO: Keep prompts concise and scoped to a single problem — describe the symptom, the root cause if known, and a short bulleted list of the specific changes needed. Aim for under ~200 words.
2. DO: Include concrete evidence — error messages, command output, or a code snippet in a fenced block — to ground the fix in reality.
3. AVOID: Long, multi-part, exploratory or evaluative prompts (e.g., broad audits, "study X", "evidence for Y", open-ended metrics work) — these correlate strongly with closed PRs. Split large asks into smaller, well-bounded tasks instead.

Historical Trends

Date	PRs	Merge Rate	Top Category
2026-09-08	1000	81.1%	bug_fix
2026-07-07	1000	80.0%	bug_fix
2026-07-06	1000	80.4%	test
2026-07-05	1000	80.8%	bug_fix
2026-07-04	1000	81.1%	bug_fix
2026-07-03	1000	81.8%	bug_fix

Trend: Merge rate has stayed remarkably stable in the ~80-82% band across all recorded runs (spanning early July through early September), with bug_fix consistently the dominant PR category by volume. No major shift observed since the last recorded analysis in July — bug-fix-style prompts remain the most reliable pattern for successful Copilot PRs in this repository.

---

Generated by Copilot PR Prompt Analysis (Run: 34276188586)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 27 AIC · ⌖ 14 AIC · ⊞ 11K · ◷

• expires on Sep 9, 2026, 12:49 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #59805.