[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-26

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-26

Summary

Analysis Period: Last 30 days
Total PRs: 637 | Merged: 508 (79.7%) | Closed: 124 (19.5%) | Open: 5 (0.8%)

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Feature Addition	214	188	22	89.5%
Testing	58	53	5	91.4%
Documentation	12	12	0	100%
Refactoring	6	5	1	83.3%
Other	57	47	10	82.5%
Bug Fix	290	203	86	70.2%

(Categorized by keyword matching on title+body; a PR can only fall into one bucket, tested in priority order bug_fix → feature → refactor → documentation → testing.)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 187.8 words (much shorter than closed PRs)
• Code block usage: 78.7% of merged PRs include a fenced code block (log excerpt, JSON, or diff snippet)
• Most common keywords: github, workflow, actions, runs, network, agent
• Structure: 41.9% use markdown headings (##) to organize the description

Example successful prompts:

1. PR Scope Copilot firewall validation to enabled web tools #63632 — "Scope Copilot firewall validation to enabled web tools": "Firewall validation was rejecting every restricted-network Copilot workflow and flagging web tools on other engines..." → Merged
2. PR Fix repo memory retry head refresh authentication #63497 — "Fix repo memory retry head refresh authentication": "Concurrent push_repo_memory writers could exhaust retries on private/internal repos because retry head refresh used un..." → Merged
3. PR Fix workflow-call SARIF artifact dependency #63495 — "Fix workflow-call SARIF artifact dependency": "upload_code_scanning_sarif referenced needs.activation.outputs.artifact_prefix without depending on activation..." → Merged

These share a pattern: symptom → root cause → concrete fix, referencing specific identifiers (push_repo_memory, upload_code_scanning_sarif) rather than abstract goals.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 336.2 words — almost double the merged average
• Code block usage: only 61.3%
• Most common keywords: operational, value, evidence, study, null, metric, graders, blocked
• A large share follow a rigid templated structure ("Record blocked operational-value study for <workflow>...")

Example unsuccessful prompts:

1. PR Record blocked operational-value study for daily-choice-test #58559 — "Record blocked operational-value study for daily-choice-test" → Closed
2. PR Record blocked operational-value study for daily-byok-ollama-test #58554 — "Record blocked operational-value study for daily-byok-ollama-test" → Closed
3. PR Record blocked operational-value study for architecture diagram workflow #58547 — "Record blocked operational-value study for architecture diagram workflow" → Closed

Key Insights

• Operational-value study PRs almost never merge. 65 PRs (10.2% of all Copilot PRs) reference "operational value/evidence/study" language; only 4 merged (6.2% success rate). PRs specifically titled "Record blocked operational-value study/design for <workflow>" were 23/23 closed (0% merge rate) — this pattern is essentially always rejected and should be reconsidered as an autonomous task type, or the acceptance criteria clarified before more are generated.
• Shorter, evidence-first prompts win. Merged PRs average 187.8 words vs 336.2 for closed — nearly 2x shorter. Prompts that lead with a specific symptom/error and a code block (78.7% of merged vs 61.3% of closed) succeed far more often than long narrative prompts.
• Bug-fix prompts have the lowest success rate among concrete task types (70.2%), likely because many describe diagnostics/investigation work rather than a verified fix, while feature/testing/documentation prompts (89–100%) tend to describe additive, self-contained changes that are easier to review and land.
• "WIP" markers correlate with worse outcomes. Of 31 PRs mentioning "WIP", only 67.7% merged — below the 79.7% overall rate, suggesting prompts producing incomplete/draft work are more likely to be abandoned.

Recommendations

1. DO: Lead prompts with the specific symptom, error message, or file/function reference, then state the fix — this pattern dominates successful merges (see PR Scope Copilot firewall validation to enabled web tools #63632, Fix repo memory retry head refresh authentication #63497, Fix workflow-call SARIF artifact dependency #63495).
2. DO: Keep prompts concise (~150–200 words) and include a concrete code/log snippet; verbose, narrative-heavy prompts correlate with closures.
3. AVOID: Generating "Record blocked operational-value study/design" PRs in their current templated form — this pattern has a 0% merge rate over 23 PRs in the last 30 days. Either change the workflow output format (e.g., a discussion or issue instead of a PR) or set clearer, narrower merge criteria.

Historical Trends

Date	PRs	Success Rate	Notes
2026-09-26	637	79.7%	Operational-value pattern rate 6.2% (65 PRs)
2026-09-25	654	79.5%	Operational-value pattern rate 21.5% (79 PRs)
2026-09-24	726	80.4%	Operational-value pattern rate 34.4% (93 PRs)
2026-09-23	763	81.6%	No operational-value tracking that day
2026-09-22	797	82.1%	Operational-value pattern rate 12.9% (70 PRs)

Trend: The overall merge rate has drifted down slightly (82% → 80%) over the past week as the volume of "operational-value study/design" PRs — a near-guaranteed-closed category — has fluctuated between 6–34% of total volume. Days with a higher share of these PRs show a lower blended success rate. The "unsuccessful keywords" list has consistently featured operational, value, evidence, study across the last 5 days of tracked history, reinforcing this as a persistent, not one-off, pattern.

---

Generated by Copilot PR Prompt Analysis (Run: 36270361406)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 35.2 AIC · ⌖ 12.8 AIC · ⊞ 10.8K · ◷

• expires on Sep 27, 2026, 12:47 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #63881.