[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-04

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-04

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 794 (79.4%) | Closed: 183 (18.3%) | Open: 23 (2.3%)
Merge rate (of completed PRs): 81.3%

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Refactoring	21	19	2	90.5%
Documentation	69	63	6	91.3%
Testing	113	98	13	88.3%
Feature Addition	164	141	18	88.7%
Other	163	141	20	87.6%
Bug Fix	464	328	122	72.9%
Security	6	4	2	66.7%

Categorization is keyword-based (first matching keyword wins); "Bug Fix" is the largest and most heterogeneous bucket, which likely explains its lower relative rate.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 166 words
• Frequent terms: fixes, workflow(s), added/adds, behavior, frontmatter, path, test(s), output
• Many successful prompts are short, targeted bug-fix reports describing a concrete symptom (e.g., "X skipped Y step, causing Z"), often quoting a specific config key, file, or error condition.

Example successful prompts:

1. PR Install AWF for firewall-enabled custom engine commands #58384: "engine.command skipped all engine installation steps, including the AWF binary required by firewall-enabled jobs..." → Merged
2. PR Refresh CI Doctor upstream source revision #58341: "CI failed because gh aw update attempted to merge CI Doctor changes from a stale upstream revision, produced conflict markers..." → Merged
3. PR [WIP] Fix failing GitHub Actions job Check workflow markdown/lock drift #58385: "Fix the f[...]" (concise, scoped bug description) → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 233 words (~40% longer than merged prompts)
• Frequent terms: null, short, keep, metric, record, label, observable, evidence, never
• A large share of closed PRs (47 of 183, ~26%) come from a single recurring template: [operational-value-study:paper-v1] ... — an automated research task asking Copilot to design a JSON "grader" schema per workflow. This template has only 1 merge out of 56 attempts (1.8% success rate), dragging down overall closed-PR statistics significantly.
• Excluding this template, the remaining ~136 closed PRs are more varied general bug-fix/feature attempts with less specific framing.

Example unsuccessful prompts:

1. PR Record blocked operational-value design for setup security audit #58543: "[operational-value-study:paper-v1] daily-action-setup-security-audit Use the repository's .github/skills/operational-value-designer/SKILL.md to design and verify one deterministic operational-value grader..." → Closed
2. PR Document blocked Craft operational-value grader design #58542: "[operational-value-study:paper-v1] craft Use the repository's .github/skills/operational-value-designer/SKILL.md..." → Closed
3. PR Record blocked operational-value study for copilot-session-insights #58541: "[operational-value-study:paper-v1] copilot-session-insights Use the repository's .github/skills/operational-value-designer/SKILL.md..." → Closed

Key Insights

• Pattern 1: The operational-value-study template prompt family has a 1.8% merge rate (1/56) vs. ~87% for all other prompts — this single recurring pattern accounts for roughly a quarter of all closed PRs in the dataset. These prompts are long, schema-heavy, and often instruct the agent to output designOutcome: "blocked" rather than ship code, which naturally suppresses merges.
• Pattern 2: Merged prompts average 166 words vs. 233 words for closed prompts — concise, symptom-first bug reports ("X does Y when Z, should instead do W") merge more reliably than long, multi-constraint specifications.
• Pattern 3: Prompts referencing a specific config key, function, or file (e.g., engine.command, gh aw update) combined with a clear before/after behavior description correlate strongly with successful merges, consistent with prior findings.

Recommendations

Based on today's analysis:

1. DO: Write concise (~100–170 word) prompts that state the specific symptom, the file/config key involved, and the expected behavior change.
2. DO: Favor feature/documentation/testing framing when possible — these categories consistently merge above 88%.
3. AVOID: Long, schema-heavy prompts with many enumerated JSON contract requirements and "blocked/rejected" escape hatches (as seen in the operational-value-study template) — these rarely result in merged code and should be redesigned or reduced in volume if the goal is landed changes rather than research records.

Historical Trends

Date	PRs	Merge Rate (completed)	Top Category
2026-09-04	1000	81.3%	Bug Fix
2026-09-03	1000	84.4%	Bug Fix
2026-09-02	1000	83.9%	Bug Fix
2026-09-01	1000	84.1%	Bug Fix
2026-08-31	994	84.3%	Bug Fix

Trend: Merge rate has dipped slightly (~3 points) from the prior 4-day average of ~84.2% to 81.3% today, coinciding with a wave of operational-value-study template PRs (56 in this snapshot, 47 closed) — a new automated research prompt family with very low success rate that is dragging down the aggregate merge rate. Bug-fix volume remains the dominant category throughout the week.

---

Generated by Copilot PR Prompt Analysis (Run: 33917319408)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 26.6 AIC · ⌖ 11.5 AIC · ⊞ 11K · ◷

• expires on Sep 5, 2026, 12:46 PM UTC-08:00