[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-27

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-27

Summary

Analysis Period: Last 30 days
Total PRs: 598 | Merged: 473 (79.1%) | Closed: 122 (20.4%) | Open: 3 (0.5%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate*
Testing	74	68	91.9%
Other	130	113	87.6%
Documentation	40	35	87.5%
Feature Addition	104	88	85.4%
Refactoring	4	3	75.0%
Bug Fix	246	166	67.8%

*Success rate = merged / (merged + closed), excludes still-open PRs.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 126.8 words — notably shorter than closed prompts
• Code blocks present: 72.9% of merged prompts
• Bullet/numbered lists present: 95.6% of merged prompts
• Top recurring terms: fixes, coding, model, workflows, safe, json, yaml, issue, outputs, actions, lock, engine

Example successful prompts:

1. PR [ci-coach] Speed up slow rate-limit sleep in cgo A-C unit-test shard #63187 — Speed up slow rate-limit sleep in cgo A-C unit-test shard: "### CI Optimization Proposal ### Summary Pre-flight validation (deps-dev, lint, lint-errors, npm-ci, build, recompile, test-unit) all pa..." → Merged
2. PR Remove push_repo_memory concurrency group so fan-out memory writes are not dropped #62414 — Remove push_repo_memory concurrency group so fan-out memory writes are not dropped: "GitHub Actions keeps at most one running and one pending job per concurrency group — every additional pending job is cancelled. push_repo_memory..." → Merged
3. PR Fix --count/--timeout wiring for multi-target logs downloads #60308 — Fix --count/--timeout wiring for multi-target logs downloads: "A multi-target gh aw logs run downloads a large number of runs but reports only a handful in the final table. This reviews the download pipeline and..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 302.2 words — more than double the merged average
• Code blocks present: 61.5% of closed prompts (less common than in merged)
• Top recurring terms: operational, value, evidence, study, null, metric, graders, blocked, assignment, adoption, rejected

A single dominant driver: 61 of the 122 closed PRs (~50%) come from the automated operational-value grader/study pipeline — these are largely [WIP] design/study PRs auto-generated to evaluate whether a workflow has a defensible metric, and many are closed by design once evaluation concludes "no defensible metric" or the design is superseded. This heavily skews the closed-PR word count and vocabulary.

Example unsuccessful prompts:

1. PR Improve AWF download failure diagnostics #63318 — Improve AWF download failure diagnostics: "AWF release-download failures could leave conclusion issues with only a generic agent failure. This adds a durable, secret-safe signal so conclusion r..." → Closed
2. PR Ensure a blank line follows closing </details> tags in generated markdown #63069 — Ensure a blank line follows closing </details> tags in generated markdown: "Generated issue/comment/discussion/PR bodies concatenate <details> blocks from several independent builders (footers, missing-info sections, diff pr..." → Closed
3. PR Fix ARC/DinD agent temporary directory staging #63045 — Fix ARC/DinD agent temporary directory staging: "ARC/DinD separates the runner and Docker daemon filesystems. The existing setup created /tmp/gh-aw/agent only on the runner, leaving safe-output pay..." → Closed
4. PR Allow add-labels max to set per-call label limit #63044 — Allow add-labels max to set per-call label limit: "safe-outputs.add-labels.max advertised limits above 10 but runtime rejected calls exceeding the fixed 10-label cap. This aligns the per-call limit w..." → Closed
5. PR Allow workflows to narrow safe-output item schemas #63016 — Allow workflows to narrow safe-output item schemas: "Safe-output tools exposed permissive built-in schemas even when workflows required stricter inputs. Invalid calls were therefore rejected only after a..." → Closed

Key Insights

• Pattern 1: Merged prompts average 126.8 words vs 302.2 words for closed prompts — more than 2x longer. Concise, tightly-scoped prompts correlate strongly with successful merges; very long prompts often indicate exploratory/study work that is closed by design rather than merged.
• Pattern 2: Bug fix prompts have the lowest success rate among substantive categories (67.8%), while testing (91.9%), documentation (87.5%), and feature (85.4%) prompts merge far more reliably — bug-fix tasks may carry more ambiguity about root cause or scope, leading to more back-and-forth or abandonment.
• Pattern 3: Nearly 50% of all closed PRs originate from the automated operational-value grader/study workflow (operational-value-study, [WIP] ... grader for ...), which is designed to close PRs when no defensible metric is found. This is an expected, structural outcome rather than a prompt-quality failure — it should be excluded or reported separately in future analyses to avoid skewing "success rate" interpretations.
• Pattern 4: Merged prompts use code blocks slightly more often (72.9% vs 61.5%), suggesting that including concrete diffs, error messages, or config snippets in the prompt correlates with successful outcomes.

Recommendations

Based on today's analysis:

1. DO: Keep prompts concise and scoped to a single, well-defined change (~100-150 words) — this is the single strongest signal correlated with merged PRs.
2. DO: Include concrete evidence in the prompt (error messages, log snippets, specific file/function references, or a small code block) rather than open-ended background narrative.
3. AVOID: Long, multi-part "study"/"design" prompts without a clear terminal deliverable — these correlate with closures, especially when the underlying task is exploratory (e.g., operational-value grader assignments) rather than a concrete code change.
4. CONSIDER: For bug-fix tasks (lowest success rate at 67.8%), add explicit reproduction steps or acceptance criteria to reduce ambiguity before dispatching to Copilot.

Historical Trends

Date	PRs	Success Rate	Top Category
2026-09-27	598	79.5%	bug_fix
2026-09-25	654	79.9%	bug_fix
2026-09-23	763	81.6%	bug_fix
2026-09-11	1000	81.1%	bug_fix
2026-09-09	1000	80.5%	bug_fix

Trend: Success rate has been gently declining over the past month (from ~81% in early September to ~79.5% today), while bug_fix remains consistently the top-volume category. The current dataset (598 PRs vs. prior windows of 650-1000) reflects a smaller 30-day rolling window rather than a drop in Copilot activity.

---

Generated by Copilot PR Prompt Analysis (Run: 36348963348)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 30.7 AIC · ⌖ 15.1 AIC · ⊞ 10.8K · ◷

• expires on Sep 28, 2026, 12:48 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #64083.