[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-06

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-06

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 806 (80.6%) | Closed: 186 (18.6%) | Open: 8 (0.8%)

Prompt Categories and Success Rates

(Categories are not mutually exclusive — a PR can match multiple keyword categories, so totals sum to more than 1000.)

Category	Total	Merged	Closed	Success Rate
Feature Addition	799	640	154	80.6%
Bug Fix	650	533	112	82.6%
Testing	524	447	73	86.0%
Documentation	339	272	64	81.0%
Refactoring	98	70	28	71.4%
Other	27	24	3	88.9%

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 143.4 words (shorter than closed PRs)
• Most common keywords: workflow, added, yaml, existing, changes, agent, path, github
• Frequent verbs: add, fix, ensure, update

Example successful prompts:

1. PR Fix Update Astro model and safe-output fallback #58909: "Fix Update Astro model and safe-output fallback" → Merged
2. PR Add compiler support for dynamic repository enclave policies #58880: "Add compiler support for dynamic repository enclave policies" → Merged
3. PR Guide Codex workflows to compatible Copilot models #58872: "Guide Codex workflows to compatible Copilot models" → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 176.5 words (~23% longer than merged)
• Most common keywords: workflow, evidence, issue, metric, direct, json, copilot
• Issues identified: closed PRs disproportionately reference evidence/metric/study-style investigative or exploratory prompts rather than concrete, scoped changes (consistent with the historical operational_value_study_template pattern, which saw ~2–5% merge rates on 2026-09-04/05)

Example unsuccessful prompts:

1. PR Expose aw_info and usage in logs JSON output #58891: "Expose aw_info and usage in logs JSON output" → Closed
2. PR Include aw_info and usage in logs JSON payloads #58890: "Include aw_info and usage in logs JSON payloads" → Closed
3. PR Include firewall details in conclusion data #58833: "Include firewall details in conclusion data" → Closed
4. PR Fix Avenger gVisor startup failure #58722: "Fix Avenger gVisor startup failure" → Closed
5. PR Record blocked operational-value study for daily-choice-test #58559: "Record blocked operational-value study for daily-choice-test" → Closed

Key Insights

• Length matters: Merged PRs average 143 words vs. 176 for closed PRs — concise, scoped prompts correlate with higher merge success. Longer prompts may signal scope creep or ambiguity.
• Testing-focused prompts perform best (86.0% success), followed by bug fixes (82.6%) and documentation (81.0%). Refactoring prompts have the lowest success rate (71.4%), suggesting refactors are riskier or harder to scope precisely.
• "Study"/"evidence"/"metric" exploratory prompts underperform: closed PRs skew toward investigative/reporting-style tasks (e.g., "record blocked operational-value study", "expose usage in logs") rather than direct code fixes — historical data shows near-zero merge rates for template-style "operational value study" prompts on recent days.
• Duplicate/near-duplicate prompts fail: PRs Expose aw_info and usage in logs JSON output #58891 and Include aw_info and usage in logs JSON payloads #58890 both attempt near-identical "expose aw_info/usage in logs" changes and both closed, suggesting redundant or unfocused prompt generation reduces success.

Recommendations

Based on today's analysis:

1. DO: Keep prompts concise and scoped to a single, well-defined change (~140 words performed best).
2. DO: Favor direct action prompts ("fix X", "add support for Y") over open-ended investigative/"record evidence" style prompts.
3. AVOID: Broad "study", "audit", or "expose metrics" style prompts without a concrete code change target — these have historically shown very low merge rates.

Historical Trends

Date	PRs	Merge Rate	Avg Words (Merged/Closed)	Top Category
2026-09-06	1000	81.3%	143.4 / 176.5	feature
2026-09-05	1000	80.1%	180.4 / 315.8	bug_fix
2026-09-04	1000	81.3%	166.1 / 232.5	bug_fix
2026-09-03	1000	84.4%	178.0 / 206.4	bug_fix
2026-09-02	1000	83.9%	215.8 / 248.3	bug_fix
2026-09-01	1000	84.1%	190.8 / 236.6	feature
2026-08-31	994	84.3%	122.3 / 143.5	bug_fix

Trend: Merge rate has drifted down slightly over the past week (84.3% → 81.3%), while the gap between merged/closed prompt length has narrowed considerably (closed-PR prompts were nearly 2x longer than merged ones on 09-05, but only ~23% longer today). This narrowing suggests prompt quality for closed PRs may be improving even as overall merge rate softens — worth monitoring over the next few days.

---

Generated by Copilot PR Prompt Analysis (Run: 34058656203)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 26.2 AIC · ⌖ 7.08 AIC · ⊞ 11K · ◷

• expires on Sep 7, 2026, 12:46 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #59299.