[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-22

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-22

Summary

Analysis Period: Last 30 days
Total PRs: 797 | Merged: 647 (81.2%) | Closed: 141 (17.7%) | Open: 9 (1.1%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Testing	436	375	87.2%
Other	22	19	86.4%
Refactoring	49	40	81.6%
Bug Fix	644	517	81.4%
Feature Addition	606	480	80.4%
Documentation	248	194	79.2%

(Categories overlap — a PR can match multiple keyword patterns, e.g. a bug-fix that also adds tests.)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 177 words (roughly half the length of closed-PR prompts)
• Most common keywords: workflow, run, add, repo, firewall, yaml, allowed
• Concrete, scoped asks tied to a single artifact (a workflow file, a CLI flag, a dependency version) rather than open-ended studies

Example successful prompts:

1. PR Add Copilot pricing metadata for grok-4.7 #62512: "The model inventory identified grok-4.7 as the only missing pricing entry..." → Merged
2. PR Update default CLI and service versions #62497: "Updates default pins for Copilot CLI, Pi CLI, GitHub MCP Server, and threat-detect, then regenerates compiled..." → Merged
3. PR Fix Daily Go Test Parallelizer terminal outputs #62494: "The workflow ran with a Codex model unsupported by its bundled runtime and could complete without emitting a s..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 320 words (nearly 2× longer than merged prompts)
• Most common keywords: operational, value, evidence, study, blocked, null, graders, domain
• A dominant, statistically strong pattern: PRs whose title/body mention "operational-value" (grader design, blocked studies, evaluator scaffolding) succeed at only 12.9% (9 merged / 70 total) — 6.5× worse than the 88.9% merge rate for all other PRs.
• [WIP]-prefixed PRs merge at only 57.1% (20/35) vs 83.3% for non-WIP PRs — indicating draft/exploratory prompts without a clear scoped fix are more likely abandoned.

Example unsuccessful prompts:

1. PR Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635: "Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs" → Closed
2. PR Add bounded issue-triage workflow #59741: "Add bounded issue-triage workflow" (operational-value study framing) → Closed
3. PR Record blocked operational-value study for daily-choice-test #58559: "Record blocked operational-value study for daily-choice-test" — this PR's own text notes the study is blocked → Closed

Key Insights

• Pattern 1 — Length inversely correlates with success: Merged PRs average 177 words vs 320 for closed PRs. Prompts that read like design documents or open studies (with evidence sections, blocked-state narratives, and speculative reasoning) close at a much higher rate than short, single-artifact fix requests.
• Pattern 2 — "Operational-value" framing is a strong failure signal: This category has held a consistently low ~12–35% merge rate across the last two weeks of historical tracking (12.9% today, 35.5% on 2026-09-12, 13% on 2026-09-11, 1.8% on 2026-09-04). These PRs tend to describe grader/evaluator design work or "blocked study" records rather than a concrete code change — reviewers appear to close them as non-actionable or superseded.
• Pattern 3 — [WIP] prefixes signal higher abandonment risk: WIP-tagged PRs merge at 57.1% vs 83.3% for the rest, suggesting exploratory/incomplete prompts are less likely to reach a mergeable state without a follow-up nudge.

Recommendations

Based on today's analysis:

1. DO: Keep prompts scoped to one concrete artifact (a specific workflow file, config value, or failing job) and describe the observed symptom + fix in ~150-200 words, mirroring the successful examples above.
2. DO: For testing/bug-fix work, cite the failing run, error message, or reproduction path directly — this category has the highest success rate (87.2%/81.4%).
3. AVOID: Framing a prompt as an open-ended "operational-value study," grader design exercise, or "record blocked state" task — these convert to merges only ~13% of the time. If a study or evaluator is genuinely needed, split it into a smaller, directly verifiable code change first.
4. AVOID: Leaving [WIP] in the PR title without a fast follow-up — these PRs merge 26 points lower than non-WIP work; convert WIP PRs to ready-for-review once the fix is concrete.

Historical Trends

Date	PRs	Merge Rate	Top Category
2026-09-22	797	82.1%	bug_fix
2026-09-21	859	81.3%	bug_fix
2026-09-13	1000	81.3%	bug_fix
2026-09-12	1000	81.0%	feature
2026-09-11	1000	81.1%	bug_fix
2026-09-10	1000	81.1%	bug_fix
2026-09-07	1000	80.4%	bug_fix

Trend: The overall merge rate has been remarkably stable at ~81% for three weeks, with bug_fix as the dominant PR category most days. The "operational-value" failure pattern is persistent — it has appeared in every tracked snapshot since 2026-09-04 with a merge rate never exceeding 35.5%, confirming this is a structural (not one-off) issue with how those prompts are framed rather than random variance.

---

Generated by Copilot PR Prompt Analysis (Run: 35781914820)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 32.2 AIC · ⌖ 7.38 AIC · ⊞ 10.9K · ◷

• expires on Sep 23, 2026, 12:49 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #63031.