[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-12

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-12

Summary

Analysis Period: Last 30 days (994 of 1000 sampled PRs fall within window; dataset capped at 1000 most recent)
Total PRs: 1000 | Merged: 810 (81.0%) | Closed: 188 (18.8%) | Open: 2 (0.2%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Testing	437	371	84.9%
Bug Fix	376	309	82.2%
Other	96	83	86.5%
Refactoring	67	53	79.1%
Documentation	232	189	81.5%
Feature Addition	619	487	78.7%

(Categories overlap — a prompt can match multiple keyword sets, e.g. "add tests" counts in both Feature and Testing.)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 176.7 words (median 152.5) — noticeably shorter than closed PRs
• Prompts read like structured bug reports: state the failure symptom first, then the fix, often with a ## Changes bullet list
• Frequent concrete references: model/CLI names (gpt-5.4, claude-sonnet-4.6), specific files, error strings (Invalid value: 'custom')

Example successful prompts:

1. PR Use a Codex-compatible model for gpclean #57051: "The gpclean workflow failed because GPT-5.4 rejected Codex custom tools with Invalid value: 'custom'..." → Merged
2. PR Use available model for Test Quality Sentinel #57815: "The Test Quality Sentinel job failed because its experiment selected claude-sonnet-4.6, which is unavailable..." → Merged
3. PR Fix Copilot SDK model inventory collection #59703: "The daily model inventory job failed because Copilot SDK 1.0.11 resolved an incompatible CLI platform package..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 315.6 words (median 272) — roughly 1.8x longer than merged prompts
• Heavy use of abstract/process language: "operational value", "evidence", "study requirements", "grader", "metric", "blocked" — these terms barely appear in merged prompts
• A distinct "operational-value study" template recurs across many closed PRs (adding deterministic graders per .github/skills/operational-value-designer/SKILL.md), asking the agent to design measurement scaffolding rather than fix a concrete bug/feature

Example unsuccessful prompts:

1. PR Record blocked operational-value study for daily-choice-test #58559: "The workflow has explicit weekday-based configuration intent but produces no durable evidence of deployment or test outcomes. Per the study requirements..." → Closed
2. PR Add operational-value grader for daily-caveman-optimizer #58557: "daily-caveman-optimizer had no deterministic operational-value grader. This adds one, following .github/skills/operational-value-designer/SKILL.md..." → Closed
3. PR Record blocked operational-value study for AWF spec surfacing #58553: "The daily AWF spec/compiler surfacing workflow lacks a direct, run-owned repository-attainment outcome..." → Closed

Key Insights

• Pattern 1 — Length inversely correlates with success: Merged PR prompts average 176.7 words vs. 315.6 words for closed PRs. Longer prompts often try to justify abstract measurement frameworks rather than describe a concrete, verifiable change.
• Pattern 2 — "Operational-value grader" prompts fail at a much higher rate: Of 93 PRs matching this template (mentioning "operational value" or "graders"), only 33 merged and 60 were closed — a 35.5% success rate, versus 85.9% for all other PRs (907 PRs, 777 merged / 128 closed). This is the single strongest negative signal found in the dataset.
• Pattern 3 — Concrete failure descriptions succeed: The most successful prompts open with a specific, falsifiable symptom (a rejected model ID, an exact error string, a failing job name) followed by a short bullet list of changes. Vague framing ("lacks a direct... outcome", "produces no durable evidence") correlates with closure.

Recommendations

Based on today's analysis:

1. DO: Lead with a concrete symptom — the exact error message, failing workflow/job name, or reproduction steps — then a short bulleted change list. This pattern dominates the shortest, most successful merged PRs.
2. DO: Keep prompts concise (under ~200 words). Success rate drops as prompt length grows past that range.
3. AVOID: Prompts framed around building measurement/evidence scaffolding ("operational value", "graders", "study requirements") without a concrete functional bug or feature — this template shows a 35.5% merge rate vs. 85.9% for everything else. If a grader/metric is genuinely needed, pair it with a specific, testable outcome rather than abstract process language.

Historical Trends

Date	PRs	Success Rate	Top Category
2026-09-04	1000	81.3%	bug_fix
2026-09-05	1000	80.1%	bug_fix
2026-09-06	1000	81.2%	feature
2026-09-07	1000	80.4%	bug_fix
2026-09-10	1000	81.1%	bug_fix
2026-09-11	1000	81.1%	bug_fix
2026-09-12	1000	81.0%	feature

Trend: Merge rate has held steady around 80–81% for the past week with no significant swing. The "operational-value grader" template first appeared as a distinct low-success cluster on 2026-09-04 (1.8% merge rate that day) and has stabilized somewhat since, but remains well below the overall average (35.5% today vs. ~81% baseline) — this remains the clearest actionable signal for prompt authors to avoid.

---

Generated by Copilot PR Prompt Analysis (Run: 34717831386)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 33.9 AIC · ⌖ 6.08 AIC · ⊞ 11.2K · ◷

• expires on Sep 13, 2026, 12:50 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #60691.