[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-24

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-24

Summary

Analysis Period: Last 30 days
Total PRs: 726 | Merged: 584 (80.4%) | Closed: 136 (18.7%) | Open: 6 (0.8%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Bug Fix	337	279	82.8%
Feature Addition	287	221	77.0%
Testing	40	37	92.5%
Documentation	13	12	92.3%
Refactoring	2	1	50.0%
Other	41	34	82.9%

(Categorized by keyword matching on cleaned agent-instruction text; a PR can only land in one category, first match wins.)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 145 words (vs. 138 for closed — length alone is not the driver)
• 78% of merged PR bodies contain a fenced code block (vs. 62% of closed) — concrete diffs/config snippets correlate with landing
• Most common keywords: github, yaml, run, coverage, changes, generated, lock, model, engine, configuration
• Short, surgical asks merge fastest — the shortest merged prompts (27–36 words) were single-purpose fixes like "Fix Copilot SDK model inventory collection" or "Use available model for Test Quality Sentinel"

Example successful prompts:

1. PR Mark Jira safe outputs as stable #62749: "Mark Jira safe outputs as stable" (27 words) → Merged
2. PR Fix Copilot SDK model inventory collection #59703: "Fix Copilot SDK model inventory collection" (35 words) → Merged
3. PR Fix Issue Monster’s unsupported Pi model #60461: "Fix Issue Monster's unsupported Pi model" (36 words) → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 138 words (comparable to merged; length is a weak signal alone)
• Only 62% of closed PR bodies include a fenced code block, vs. 78% for merged
• Dominant, closed-skewed vocabulary: operational, evidence, study, intent, adoption, metric, attainment, record, assignment, design, deterministic, baseline, grader — these terms cluster almost entirely around one recurring initiative
• Single biggest driver of closures: PRs referencing "operational-value" grader/study work (93 PRs, only 32 merged = 34.4% success rate, vs. 88% for everything else). Titles like "Record blocked operational-value study for X" or "Add operational-value grader for Y" account for a disproportionate share of the 136 closed PRs — many are explicitly recorded as "blocked" by design and closed intentionally rather than due to a bad prompt.

Example unsuccessful prompts:

1. PR Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635: "Fix operational-value grader created-at acquisition and preserve grader result contract..." → Closed
2. PR Add operational-value grader for daily-caveman-optimizer #58557: "Add operational-value grader for daily-caveman-optimizer" → Closed
3. PR Record blocked operational-value study for daily-byok-ollama-test #58554: "Record blocked operational-value study for daily-byok-ollama-test" → Closed
4. PR [WIP] Add deterministic operational-value grader for code-simplifier #58522: "[WIP] Add deterministic operational-value grader for code-simplifier" → Closed

Key Insights

• Pattern 1: The "operational-value grader/study" workflow initiative accounts for ~68% of all closed-not-merged PRs (93 of 136) despite being only ~13% of total volume — these are largely intentionally recorded as blocked/incomplete evaluations, not failed coding attempts. Excluding this cluster, the underlying success rate across genuine feature/bug-fix work is closer to 88–90%.
• Pattern 2: Presence of a fenced code block in the prompt/PR body correlates with a ~16pp higher merge rate (78% vs 62%), suggesting prompts that show or reference concrete code/config content guide the agent to a landable change more reliably than prose-only asks.
• Pattern 3: Prompt length is not a strong standalone predictor this cycle (145 vs 138 words) — specificity and scope (single, concrete target) matters more than verbosity. The very shortest merged prompts were tightly scoped one-line fixes.
• Pattern 4: WIP-tagged PRs ([WIP] in title) merge at 67% (22 of 33) — lower than the overall 80% baseline, consistent with WIP marking genuinely unfinished or exploratory work.

Recommendations

Based on today's analysis:

1. DO: Scope prompts to a single, concrete fix or feature (file/function/behavior named explicitly) — the shortest, most surgical merged prompts had the fastest, cleanest outcomes.
2. DO: Include a code snippet, config excerpt, or command output in the prompt/issue body when possible — PRs with a fenced code block merge ~16pp more often.
3. AVOID: Open-ended "study"/"design"/"grader" exploratory tasks without a clear acceptance criterion — this category alone drove most of this period's closures, largely by design (recorded as "blocked"), so treat it as a distinct, expected-low-merge-rate workstream rather than a prompting failure.

Historical Trends

Date	PRs	Merge Rate	Notable
2026-09-24 (today)	726	80.4%	opval cluster still dominates closures (34.4% success)
2026-09-23	763	80.7%	avg closed-prompt length spiked to 285 words
2026-09-22	797	81.2%	opval pattern: 9/70 merged (12.9%)
2026-09-21	859	81.3%	opval_template: 8/68 merged (11.8%)
2026-09-13	1000	81.3%	avg closed-prompt length 343 words
2026-09-08	1000	81.1%	code-block presence: 83.5% merged vs 65.4% closed

Trend: The overall merge rate has been remarkably stable (~80–84%) across the whole tracked history. The recurring "operational-value" grader/study initiative continues to be the dominant, structurally-distinct source of closed PRs (10–15% success rate each cycle), while all other categories consistently land in the 75–95% range. Sample size has been shrinking slightly cycle over cycle (1000 → 726) as the initial 30-day window rolls forward past a historically high-volume period.

---

Generated by Copilot PR Prompt Analysis (Run: 36056694361)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• o205451.ingest.us.sentry.io

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "o205451.ingest.us.sentry.io"

See Network Configuration for more information.

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 43.5 AIC · ⌖ 7.28 AIC · ⊞ 10.9K · ◷

• expires on Sep 25, 2026, 12:50 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #63511.