[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-23

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-23

Summary

Analysis Period: Last 30 days
Total PRs: 763 | Merged: 616 (80.7%) | Closed: 139 (18.2%) | Open: 8 (1.0%)
Merge rate (of completed PRs): 81.6%

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Bug Fix	654	525	122	81.1%
Feature Addition	95	80	14	85.1%
Testing	6	6	0	100.0%
Other	5	4	1	80.0%
Documentation	2	1	1	50.0%
Refactoring	1	0	1	0.0%

Note: Categorization is keyword-based (title + body); "bug_fix" dominates because most Copilot PRs in this repo are workflow/compiler fixes. Testing/documentation/refactor have very small samples (n≤6) and their rates are not statistically robust.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 138.5 words (notably shorter than closed PRs)
• Most common keywords: workflow, fixes, run, generated, actions, coverage, changes
• Merged prompts tend to name a specific failing workflow, job, or error condition up front, then describe the fix concisely.

Example successful prompts:

1. PR Use portable path for OTLP helper guidance #62940: "OTLP examples hardcoded /tmp, which is not portable across runner environments. Update maintained guidance to resolve..." → Merged
2. PR Allow line inspection in PR code quality reviews #62770: "The PR Code Quality Reviewer failed after repeated permission denials for nl and sed, causing the Copilot harness to..." → Merged
3. PR Fix Daily Code Metrics Copilot model configuration #62764: "The Daily Code Metrics workflow was failing at runtime because the Copilot engine requested an unsupported bare model na..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 285.4 words — roughly 2x longer than merged prompts
• Most common keywords: workflow, run, generated, workflows, lock, output, validation, json
• Closed prompts more often reference multiple files/subsystems (lock, json, validation, output together), suggesting broader, less-scoped asks that may require follow-up iterations the agent couldn't resolve alone.

Example unsuccessful prompts:

1. PR Fix Cursor smoke workflow telemetry failure #62815: "The Cursor smoke workflow failed after successful agent runs because its shared telemetry check expected AWF API-proxy t..." → Closed
2. PR Prevent cross-engine version reuse in threat detection #62797: "Smoke Goose's detection job passed Goose version 1.45.0 to the Copilot CLI installer, causing a 404 for a nonexistent..." → Closed
3. PR Add opt-in shell-free Go repository profile for Copilot SDK #62776: "Copilot SDK workflows lacked a closed native repository lifecycle when shell and task tools were disabled. This adds an..." → Closed

Key Insights

• Conciseness correlates with success: Merged PR prompts average 138.5 words vs. 285.4 for closed PRs — nearly 2x shorter. This matches the trend seen across recent daily snapshots (e.g., 2026-09-22: 177 vs 320 words), suggesting tightly scoped, single-issue prompts are more likely to land.
• Bug-fix framing dominates and performs solidly: 654 of 763 PRs (86%) are bug-fix style prompts (naming a specific failure/error), with an 81.1% merge rate — consistent with historical averages (80–84% over the past two weeks).
• Feature-addition prompts slightly outperform bug fixes (85.1% vs 81.1%) in this window, though sample size (95) is much smaller than bug-fix (654); this differs from some earlier periods where bug fixes led, so treat as a soft signal rather than a hard rule.
• Multi-subsystem prompts underperform: Closed PRs frequently reference several cross-cutting concerns (workflow, lock file, JSON output, validation) simultaneously — a proxy for scope creep. Scoping a prompt to one specific symptom + one fix location appears to correlate with merge success.

Recommendations

Based on today's analysis:

1. DO: Keep prompts concise (~100–150 words) and lead with the specific symptom or error message observed, plus the exact workflow/component affected.
2. DO: Scope each prompt to a single fix or feature; when multiple concerns exist (e.g., a broken lock file and a validation bug), split them into separate PR requests.
3. AVOID: Long, multi-paragraph prompts that bundle several unrelated changes (schema updates + wiring + tests + docs) — these are more common in closed PRs.

Historical Trends

Date	PRs	Merge Rate	Top Category
2026-09-23	763	81.6%	bug_fix
2026-09-22	797	82.1%	bug_fix
2026-09-21	859	81.3%	bug_fix
2026-09-13	1000	81.3%	bug_fix
2026-09-12	1000	81.0%	bug_fix
2026-09-11	1000	81.1%	bug_fix
2026-09-10	1000	81.1%	bug_fix
2026-09-07	1000	80.4%	bug_fix

Trend: Merge rate has been remarkably stable in the 80–82% band over the past 2+ weeks, with a slight uptick on 2026-09-22 (82.1%) before settling to 81.6% today. bug_fix remains the dominant and consistently top-performing category by volume; the average prompt-length gap between merged and closed PRs (merged shorter) is a persistent pattern across recent snapshots.

Data Quality Notes

• 8 PRs are still Open and excluded from success-rate calculations; they may shift outcomes once resolved.
• Categorization uses keyword matching on title+body text; some PRs may be miscategorized, and categories other than bug_fix/feature have very small sample sizes (n≤6) in this 30-day window, limiting statistical confidence.

---

Generated by Copilot PR Prompt Analysis (Run: 35917583833)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• o205451.ingest.us.sentry.io

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "o205451.ingest.us.sentry.io"

See Network Configuration for more information.

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 27.7 AIC · ⌖ 6.35 AIC · ⊞ 10.9K · ◷

• expires on Sep 24, 2026, 12:47 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #63256.