[prompt-analysis] Copilot PR Prompt Analysis - 2026-11-10

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-11-10

Summary

Analysis Period: Last 30 days (2026-10-11 → 2026-11-10)
Total PRs: 612 | Merged: 512 (83.7%) | Closed: 83 (13.6%) | Open: 17 (2.8%)
Merge rate of completed PRs: 86.1% (512 / 595). No PRs had an empty body.

Prompt Categories and Success Rates

Categories are assigned by keyword matching on the PR title. Success rate = merged / (merged + closed).

Category	Total	Merged	Closed	Success Rate
Bug Fix	70	57	8	87.7%
Feature Addition	76	61	14	81.3%
Refactoring	52	42	10	80.8%
Documentation	38	33	4	89.2%
Testing	33	29	3	90.6%
Other / uncategorized (route, bump, clarify, ...)	343	290	44	86.8%

Refactoring and feature prompts are the weakest categories. Test and docs prompts are the strongest, but those samples are small (33 and 38 PRs).

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 172 words (median 101)
• Prompts that name a concrete file or path merge at 93.2% (162 PRs), against 83.4% for prompts with no file reference (433 PRs)
• Prompts of 200 words or more merge at 92–97%. Prompts of 100–200 words merge at 82.4%.
• Keywords skewed toward merged PRs: unchanged, only, routed, provider, test, follow, step
• Title verbs with a 100% merge rate: Route (38), Bump (14), Clarify (12), Update (9), Record (9)

Example successful prompts:

1. PR Run NVX musl integration selectively on PRs #76277 "Run NVX musl integration selectively on PRs": "The NVX musl inference job runs for unrelated CJS changes. Gate it on relevant PR paths without chan..." → Merged
2. PR Diagnose and release stalled code-scanning fixer attempts #76184 "Diagnose and release stalled code-scanning fixer attempts": "Stalled fixer attempts lacked a clear diagnostic and could remain unavailable for retry. The workflo..." → Merged
3. PR Support name-based artifact upserts in update-issue #75988 "Support name-based artifact upserts in update-issue": "Issue artifacts were limited to newly created issues. This change lets update_issue publish artifa..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 138 words (median 93)
• Keywords skewed toward closed PRs: changes, work, reported, configuration, retain, preserving, findings
• Issues identified:
  • [WIP] placeholder PRs: 9 closed, 3 merged and 5 still open. The merge rate for completed ones is 25%, against 87.3% for non-WIP PRs.
  • Investigate/verify/triage prompts that end in "no changes": at least 5 closed PRs concluded that the work was already done or the finding was a false positive. Examples are Verify ZIP extraction bounds and scanning status #76434, No changes: decorative logo accessibility is already covered #76163, Triage 2026-11-04 security scan findings #75621 and Triage unsubstantiated red-team findings #71621. Prompts of the form "Verify/Triage/Investigate ..." had 8 closed PRs in the title sample.
  • Under 50 words: 75% merge rate (12 PRs), the lowest length bucket.
  • Checklist-style bodies (- [ ]): 54.5% merge rate, but only 11 PRs, so this is weak evidence.
  • Threat-detection error PRs: 2 closed PRs carried a threat-detection engine error banner (e.g. [review] Remove blank line before generated footer expiration #74211).

Example unsuccessful prompts:

1. PR Verify ZIP extraction bounds and scanning status #76434 "Verify ZIP extraction bounds and scanning status": "Validated the existing ZIP extraction protections and their regressions; no source changes were need..." → Closed
2. PR Triage 2026-11-04 security scan findings #75621 "Triage 2026-11-04 security scan findings": "The scan reported 175 findings across setup scripts, but the issue provides only candidate filenames..." → Closed
3. PR Enable readerstringconv in native and wasm CI #72096 "Enable readerstringconv in native and wasm CI": "readerstringconv was not CI-enforced, leaving three production calls that copied byte slices into..." → Closed

Key Insights

• Concrete file references help. Prompts that name specific files or paths merged 9 points more often than those without (93.2% vs 83.4%).
• Investigation-only prompts rarely produce a mergeable diff. "Verify", "Triage" and "Investigate" tasks often end as "already covered" or "false positive" and are closed. Ask for a defined change, or have the agent report findings in an issue instead.
• Longer, detailed prompts hold up. 200+ word prompts merged at 92–97%, while very short prompts (under 50 words) were the weakest bucket (75%). Within the 50–200 word range, longer was not clearly better.
• Weekly volatility. The merge rate by creation week was 83.0%, 89.5%, 76.3%, 91.2% and 87.8% (n = 147, 76, 97, 193, 82). The dip in week 3 (starting 2026-10-25) coincides with a batch of triage and [WIP] closures.

Recommendations

1. DO: Name the target files, functions and expected behavior (for example "route X to Y in pkg/..."). Narrow, mechanical tasks such as route, bump, clarify and update merge at ≈100%.
2. DO: State an acceptance criterion (test to add, behavior to preserve or leave unchanged). unchanged and only correlate with merges.
3. AVOID: Open-ended verify, triage or investigate prompts that can legitimately resolve to "no change needed". Pre-check whether the work is already done before assigning it.
4. AVOID: Leaving [WIP] placeholder PRs open. They are the most likely to be closed (25% merge rate).
5. AVOID: Very short prompts. Add at least the file, the symptom and the desired outcome.

Historical Trends

Date	PRs	Merge Rate (of all)	Avg words merged / closed
2026-11-10	612	83.7% (86.1% of completed)	172.0 / 138.0
2026-11-08	589	85.4% (86.7% of completed)	173.4 / 141.3
2026-11-02	563	84.4% (84.8% of completed)	164.7 / 141.7
2026-10-28	606	83.8% (84.1% of completed)	160.3 / 137.9

Trend: The merge rate of completed PRs has held at 84–87% for a month. The window has moved from about 1000 PRs in the summer to about 600 now. Average merged prompt length has crept up from about 160 to about 172 words, while closed prompts stay shorter at about 138.

Method note: Categories come from title keywords, so 56% of PRs fall in "Other". Small categories (n < 40) and single-digit closed counts should be read as directional, not statistically significant.

---

Generated by Copilot PR Prompt Analysis (Run: 42897215940)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · routed: sonnet55 high · ~24 AIC · ⌖ ~2 AIC · ⊞ 6.2K · ◷

• expires on Nov 11, 2026, 12:44 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-11-11T20:44:44.851Z.

Closed by Workflow