[prompt-analysis] Copilot PR Prompt Analysis - 2026-07-03

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-07-03

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 801 (80.1%) | Closed (not merged): 178 (17.8%) | Open: 21 (2.1%)

Overall success rate (merged / completed): 81.8% — stable, up from 81.8% on 2026-07-01.

---

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Testing	17	15	2	88.2%
Documentation	15	13	2	86.7%
Refactoring	17	14	3	82.4%
Feature Addition	282	228	51	81.7%
Bug Fix	650	516	116	81.6%
Other	19	15	4	78.9%

Dominant category: Bug Fix (65% of all PRs)

---

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: ~186 words
• Top keywords (relative frequency): workflow, actions, runs, agent, output, safe, test, json
• Strong use of problem context: merged prompts clearly explain what broke or what is missing before describing the fix
• Code/file references: successful prompts name specific packages, files, or fields (e.g., pkg/modelsdev, compiler_types.go)

Example successful prompts:

1. PR fix: remove gpt-5.5 from agent model alias #43229 (fix: remove gpt-5.5 from agent model alias) → Merged

   "gpt-5.5 does not support the /chat/completions endpoint used by the Copilot SDK, causing CAPIError: 400 failures in any workflow where the agent model is...`"

2. PR Refactor main-job runtime setup orchestration to reduce largefunc backlog #43105 (Refactor main-job runtime setup) → Merged

   "This PR takes one focused slice of the largefunc backlog in pkg/workflow by reducing function length in compiler_yaml_main_job.go without changing..."

3. PR Quick Start: define frontmatter inline and surface .lock.yml semantics as a callout #43083 (Quick Start: define frontmatter inline) → Merged

   "Quick Start used frontmatter without enough in-context definition for first-time users, and the .lock.yml explanation was easy to miss in plain body..."

---

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: ~183 words (similar to merged — length alone is not a predictor)
• Top keywords relative to merged: model, issue, shared, updated, outputs
• Prompts that address already-superseded solutions (e.g., deduplication was fixed via a different approach)
• Competing PRs: multiple PRs tackling the same bug from different angles — only one merges
• Overly broad scope: PRs touching 53+ workflows at once tend to be replaced by narrower targeted PRs

Example unsuccessful prompts:

1. PR Migrate create-issue deduplicate-by-title to templatable boolean and add integer codemod #43086 (Migrate create-issue deduplicate-by-title to templatable boolean) → Closed

   "safe-outputs.create-issue.deduplicate-by-title was effectively on the wrong contract: the compiler/runtime path still supported integer distance values..."

2. PR Refactor 53 workflows to use shared reporting-otlp import bundle #42978 (Refactor 53 workflows to use shared reporting-otlp import bundle) → Closed

   "53 workflows were still importing shared/reporting.md and shared/otlp.md separately..."

---

Key Insights

• Specificity wins: Prompts referencing concrete identifiers (function names, package paths, error codes) consistently outperform vague descriptions. Testing and Documentation categories, which tend to have clear, bounded scopes, top the success rate table.
• Scope matters more than length: Prompt word counts are nearly identical between merged (186) and closed (183) PRs. The difference lies in scope — PRs closed for duplication or supersession often had larger change surfaces (e.g., touching 53 files), not longer prompts.
• Bug-fix volume dilutes success rate: Bug Fix is by far the largest category (650 PRs, 65% of total) and drives the overall rate. When multiple Copilot PRs compete to fix the same bug, the closed ones look identical in quality — they just lost the race.

---

Recommendations

1. DO name specific files, functions, packages, or error messages in the prompt — this creates a tighter task scope and reduces overlap with competing PRs.
2. DO prefer test and documentation prompts when possible — these categories show the highest success rates (88% and 87%) and are least likely to produce competing PRs.
3. AVOID bulk-migration or cross-cutting refactor prompts that touch many files in a single PR — these are the most common closure pattern and are typically superseded by narrower follow-ups.
4. AVOID submitting duplicate bug-fix prompts for the same issue without first checking if an open Copilot PR already addresses it.

---

Historical Trends (last 7 runs)

Date	PRs	Merged	Closed	Success Rate
2026-06-25	1000	801	192	80.7%
2026-06-26	1000	800	186	81.1%
2026-06-27	1000	809	187	81.2%
2026-06-28	1000	807	188	81.1%
2026-06-29	1000	809	185	81.4%
2026-06-30	1000	814	180	81.9%
2026-07-01	1000	812	180	81.8%
2026-07-03	1000	801	178	81.8%

Trend: Success rate has been remarkably stable at ~81-82% over the past week. Closed PRs have been trending slightly downward (192 → 178), indicating fewer superseded/competing PRs. Open PRs spiked today to 21 (vs. 6-8 recently), suggesting a burst of new activity yet to be reviewed.

---

References:

• §28683033856

Generated by 🔍 Copilot PR Prompt Pattern Analysis · 47.3 AIC · ⌖ 13.3 AIC · ⊞ 7.8K · ◷

• expires on Jul 4, 2026, 1:04 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #43434.